The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot.
As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.
This is what makes open weight models so threatening to them. The political and “it’s China” angle is mostly just a cover for the real reasons why they’re freaked out.
The fact that models are now a pure commodity is bad enough for the big labs. If small open weight models become the norm the big labs are toast.
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages.
Most generic use cases do benefit from a model that has been trained broadly. When you don’t know the specific use case ahead of time, you have to have world knowledge ready to go. Even when coding it’s helpful to have all that knowledge on tap so the model can understand product intent and use cases for the product you’re building.
> As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles
The real expensive part of fine tuning is gathering a good data set. That’s been the hard part of training anything for a long time. If you’re lucky enough to have a neatly organized and clean data set then you can attempt it, but you need to be in a position to run evals and measure quality.
I’ve done it, but there are so many use cases where the engineering, data labeling, and ongoing quality review hours cost so much that it would be cheaper to continue using a frontier lab model that just works from the start.
Can't you take a broadly trained model and throw away useless knowledge (like foreign languages, and 99% of wikipedia) and still end up with a model that is as intelligent and useful as the original but much smaller?
Models are not a collection of training data. The entire model is shaped by the training data it sees during training.
What we call knowledge is encoded as weights. Something we identify as knowledge is the sum of many weights across the model. However the same weight may be involved in very different topics. You can’t point to every weight and say that this one is only for a part of Wikipedia or Harry Potter that I don’t need. That weight could be active for a multitude of topics.
There have been a lot of attempts to reduce the size of mixture of experts models by selectively removing experts. The results are not good, but knowledge isn’t neatly separated within the model.
English reasoning is far better when the model has been trained on many other languages even if you only work in English and never touch another language.
I'm not an AI researcher (would love to hear an actually informed answer on this!), but AFAIK the weights are more or less a black box - you can't easily pinpoint that weights x, y and z relate to Ancient Rome and can therefore be safely removed from a model that's optimised for coding.
I suspect it's a bit like tree-shaking/dead-code elimination in an untyped, dynamic language. You can probably delete a lot of things, but each deletion has a chance to break (or silently degrade) some functionality. Good coding practices, good docs and comments, and good tests can help determine whether a given definition is safe to delete or quickly verify that the deletion was a mistake after the fact. LLMs' weights lack all of those. It's like trying to perform DCE on a binary linked from outputs of ten different compilers with different optimization levels (and the linker is outright bonkers) without running the binary once. Oh, and the binary is many tens of GB in size. Yikes, I'm getting goosebumps just imagining it!
if you have the model weights or a provider that will give you the full token distributions you can have the next best thing and do a domain-specific distillation. this way you can take a 9B or 27B LLM and uplift it to the level of the larger model.
it still takes some work and ingenuity, you need to design a workflow that would let you efficiently probe the larger model with your domain-specific prompts, this usually involves a workflow where you obtain the inverse mapping from frontier models (don't need distributions for this step).
Furthermore, shifting the complexity to the model simplifies engineering. A lot of companies made this trade, replacing their bespoke ML models with foundation models.
The premise of your project seems compelling. Is it novel or building on existing work? Anything one could read or watch to get introduced to that area of research?
You would already be able to see RSI and loops being a thing.
So there's going to be more and more research coming out telling you what size of models are really needed to solve a specific task. When you combine that with RSI, the next frontier would be to get this model as close to the real world with instant data and instant reasoning, no fine tuning at all. And see how this one could iterate itself.
I'll also be trying out more tiny models and seeing if they can call each other, and be as effective; You will also be seeing me trying this, in addition to what I'd already mentioned in the other reply.
I did some research about scaling laws and what's unsolved there.
This project was built on structural sparsity, do away with any requirements to train models on largely outdated corpus of data, and only train it on reasoning logic. Because if models can use tools well and retrieve everything else, it greatly would reduce the need for large language models.
Further, when generating traces, I ran a benchmark that showed that on grounded data, n-grams and simple retrieval outperformed transformers on efficiency (See https://github.com/guilt/Transformers) and wanted to try scaling this up to much larger traces.
It worked quite well. Next improvement would be to run this with a BERT model and add more do-not-know traces and the Claude style neurotic self-doubt/self-verify traces and test it out a bit more.
If the entropy required to route people to the relevant answers is very low, full dense attention is a waste of compute. That's the theory being pressure tested here. The more I do this exercise, the more I am getting convinced that smaller models are good enough.
I took a look at the codebase and found that this is just pattern matching. There is no any novelty here. It looks good for knowledge-base retrieval system, but other than that, this is just nothing more than a pattern-matcher.
Would love to be proven wrong with future updates.
Why don't you go and actually build something novel? People need to understand better how current models work, and if retrieval indeed solves 99% of the problems people care about, then those other models aren't doing anything super novel either.
I'm not arguing here: The code is out there, data is out there. Do better if you know something actually better. Thanks!
Right now, we keep creating new models with larger and larger weights and throwing hardware at it. His argument is that this is a dead end and ultimately we'll eventually go back to purpose fit algorithms like we've always done in the history of AI development.
Did you even watch the video? Purpose fit algorithms outperform is quite literally the OPPOSITE of Suttons argument. This is the man who wrote this:
“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”
His problem with LLMs are they aren’t general enough… In fact his essay the bitter lesson was what inspired the current LLM scaling.
If you watched the video, you can even see the part where he talks about how AI will fully replace humans as the next stage of life. How wholesome!
Sure, for business, personal, etc it makes sense to build purpose built but that’s not what these companies are trying to achieve in the end. They want to own a nuclear weapon that outstrips everything else as soon as it exists so they “win”
As someone who worked at multiple startups, I am pretty sure they get it but it doesn’t fit their goals.
They want to moat where they don’t need to compete with other companies because they have something that other companies can’t have.
In my opinion this is a short-sighted and greedy worldview. Haven’t seen it work personally. It is a different version of the month-to-month salary guy thinking he will be a billionaire and having that thrash “mindset”.
The reality is that practically none of those companies will amount to anything and they would be better off weighing the usefulness aspect of their output more. Instead they are imagining they will be Google.
Anthropic and openai ofc are the pinnacle of this greed culture and they correspond to FTX from the crypto trash hype so I don’t think they fit into the scale of sensibility.
Coming from this perspective, it is pretty easy to see what they are.
Going by the chart in the article, if your total workload is 1k cataloged items and your quality threshold is 70%, why wouldn't you just pay $19 to gemini API instead of $500 + time to make a custom fine tune?
The resultant model much cheaper per call and outperforming frontier models. So you'd quickly make your money back at any sort of meaningful scale, continue to pay less from that point on, and have better results. And I think removing a third party dependency is also a benefit in and of itself.
But then can't the case be made that narrower and stricter-defined use cases are better served by more conventional ML? If/Wherever efficiency is a concern, that is.
LLMs fit between well defined and poorly defined data; if you have well defined data, you can just feed it to ML models and they'll do what you want.
LLMs will take poorly defined data and "potentially" create well defined output.
So ideal systems will likely be constructed with a LLM on one end and a ML on the other and some how, if you can feedback poor data from the ML back into the LLM to clean up the data, you have a magic layer that doesn't care as much about the structure of the data.
That's a lot of supposition, but that's the difference I see between what we can do with LLMs and how ML models are structured. They might be classified as doing the same thing, but their data inputs are vastly different.
Yeah and these latest and greatest models universally suck hard for any use case outside of the few 'blessed' ones. Like I'm sure their ability to write prose and generally sound like a human being has regressed quite a bit, but even if not. Opus 5 is barely above GPT4 when it comes to stuff like home improvement advice.
Fine timing takes time and data, though. If smart enough models get cheap enough, then most people have lots of use cases that are cost insensitive enough that it's not worth the effort.
Of course, "smart enough" is a low enough threshold for most uses that this is still a problem for the frontier labs.
But at the same time, a truly smart enough closed model could also potentially command almost whatever they'd care to charge for it.
Whether they can actually get to that level remains to be seen, but I can definitely see a situation where most people are perfectly happy with cheap middle of the tree models while large corporations pay magnitudes more than current API pricing for access to models never even marketed as a mass market product and keep the labs afloat.
It's of course be a lot easier for them to find the path towards that of they didn't need to compete with open models in the meantime.
Yes. Fine tuning is an optimization. Like all optimizations, it’s downstream of figuring out your problem, implementing a naive solution, scaling the naive solution, and getting frustrated by SWAP-C. THEN you start poking at what optimizations are possible, choosing an approach, implementing the optimization, and redeploying.
This cycle happens when you have a well defined use case with high volume. It is the far opposite end of the spectrum from the general purpose intelligence on tap that frontier AI models purport to deliver.
I think focused fine tunes and big general models will coexist. Ideally with smart routing and caching to use small, specialized and local options when appropriate.
I mostly agree, but at the same time I wouldn't blindly ignore the "it's China" angle at all. These are models you cannot open up and do a thorough audit of. I do think local compute will eventually be good enough that most people will just use local models. Microsoft might be the ultimate winner of that if they can stop ruining Windows and focus on making an OS similar to how straightforward and simple macOS is, no ads literally everywhere for Microsoft Office. Just make a good OS Microsoft, is that so much to ask? You HAD a really solid OS and you ruined it.
>As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.
I have a genuine question and I'd like to hear people's good faith thoughts on this.
There's a fair case that open models are threatening to institutions who spent a lot on training proprietary SOTA models.
But to my understanding, the massive investment spend (much of it debt backed as you note) is on data centers, chips, physical infra.
Yes, this infra is needed to train the models, but it is also needed to serve inference.
Perhaps the costs associated with training SOTA models is ultimately a "bust" given open models eroding the SOTA closed model performance advantage.
But demand for inference is skyrocketing and there seems to be no end in sight.
The physical hardware underpinning inference is in fact a scarce good (currently, and this seems sustainable at least over mid-term). And inference is a scarce service as such.
I know that cost of inference constantly goes down as models, technical infrastructure, and applied AI techniques become more efficient (specialized SLMs etc). So this puts downward pressure on prices.
But still... demand for inference is just growing like crazy regardless. Putting upwards pressure on prices.
Doesn't this mean that all the spending on AI infra is much better positioned to get positive ROI regardless of the type of model being served?
Put another way, models seem to be commoditizing, but physical hardware is not (currently).
The vast majority of the AI boom spend is on hardware to my understanding (even training capex can be repurposed for inference).
Doesn't this suggest that the economics for the "railroads level of build-out spending" are healthier than they might seem at first glance?
Maybe. But you have to consider any difference in expected returns. Also account for expected efficiency and capability advances of new hardware. And perhaps most importantly, as of a few years ago the failure rate of these GPUs when run hard was in the ballpark of 10% per year (that might or might not change but regardless it definitely needs to be accounted for).
It seems exceedingly unlikely that we will see growth sufficient to justify the level of investment. But as you say that doesn't necessarily mean a worst case scenario either. An investment can be bad without being ruinous after all.
seems that, like software engineering and other areas before, they also have to rediscover that one single monolithic solution that handles everything is too inflexible and not maintainable, it's just a bad approach. people don't need the models they use to generate their codebase to also be able to translate Shakespeare into gen-z slang
Im pretty sure they know quite well but its not their current focus on creating or allowing others to create small finetuned and optimized models.
In the race they are in, its still highly beneficial to be the frontier model.
I use the frontier model every single day through my company and my company happily spends these tokens.
It makes a huge difference if different people can use one interface to do everything.
Big models give you fundamental things: A lot of facts/context, usability (you might be a native english speaker, don't underestimate how hard it is for A LOT of people to formulate what they want/need in english only AND a low complexity.
No one needs to build a router and x sub models and a router architecture. You literaly just have an API, you might choose the model and the effort but thats it.
I find this current state of the art a LOT more telling on the current progress we are in than anything else. I'm confident that small optimized models will become a lot more relevant like lets say java + english + a second language + spring boot + postgresql. It might also be beneficial in the long term to finetune with your project details.
For a lot of very technical non human interfacing things, finetuning is happening left and right.
But what i find very interesting is emerging complexity capabillity. I believe that fables skill to hold more topics and combine more complex solutions together is because of its parameter size.
We will have to figure out if we can extract this complexity out of it while reducing the training data in a way that the training data focus more on thinking. Plenty of smaller thinking models show that this is doable.
Btw. Mixture of Experts is for sure not optimal for this, but it already is a form of optimized sub models. Perhaps we might just have MoE with a million experts in the future. One per lanuage + area of expertise etc.
Also don't forget: IF AGI is coming through a current frontier model, you will let it work for hours, days and weeks on one problem completly independent of any human input and it will be better than a human. If they reach this before a collapse, we are done and they 'won'. For this you need big frontier models.
I’m not sure it will go this direction. Using a generic (near) frontier model is often cheap enough that you need to talk really big volumes before it pays off to fine tune.
My example: we were doing single digit millions of automated call summaries a few years back at a major bank with GPT-4o. Smaller model gave more rejected summaries (compliance not happy), so we briefly looked at fine tuning a smaller model and basically concluded that even at that scale the effort of data collection, management, fine tuning, hosting the model, etc didn’t have a sufficient business case vs picking up other projects.
I actually prefer using less powerful models most of the time, I might use a stronger model initially and then switch to weaker models as I fine tune the output.
Offloading too much of your task to models eliminates the human ownership, without ownership you can't move forward. Context sizes can't keep up with large codebases and markdown files with instructions and guidelines only take the LLM so far in the ownership aspect.
My gut feeling is that to start offloading ownership to the LLM we would need to see at least two order of magnitude increases on context size.
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages.
I think that's true across the board. Along those lines, most companies all of a sudden started looking for AI researchers, instead of software engineers, which is what they still need :-).
Just like a decade-ish ago, when the FAANG interview style got openly known, then everybody started to conduct interviews in that pattern :-)
I would not say cheap per say, but in the context of "where is the business success going to be" that's one of the reasons why Mistral focuses on providing tuned model on premises to their customers.
Some specialized models may end solving themselves by helping fit the problem with the appropriate regular algorithms/formulas, which are a million times more efficient. So they are probably less attractive to dump money on. As of currently they still benefit from expressing lots of patterns that nobody bothered formalizing.
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages
I think they understand it but they think they can get away with it because they own the narrative. As long as they can make people believe they need such a model, it doesn't matter if it's true or not.
They are playing the cloud playbook, it didn't matter that most companies didn't need 99,999% uptime and instantaneous horizontal scaling, as long as people believed they did they are happily paying 10-100x the cost to AWS instead.
Every time I see this kind of story, two things bother me.
First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining your fine-tune.
Second, the $500 training bill is the cheapest line item in this story. The expensive parts are creating the data and maintaining the model afterwards. How many use cases can actually produce 177k scored episodes? Here they had to generate them synthetically from Amazon Berkeley Objects. To me, that dataset is the strongest evidence in the article of how hard fine-tuning is to apply: if the data existed naturally, nobody would need to manufacture it.
Regarding your first point, if the cost of not doing it while you wait exceeds the cost of training/maintaining the model, then this approach still makes sense. This really depends on how fast you expect cheap (and open-source) models to improve. So it might only be a temporary strategy, but still worthwhile. Also, I expect that specialized models will always be better (cheaper or better outcomes) than a general model, similar to how specialized HW like GPUs is still used even though CPUs have improved a lot too.
Maintaining in the way of ongoing training is relatively cheap, considering that the initial training run is only 500$ (in this example). Creating new examples to account for drift in the training data is more expensive, but these examples can then be reused when training a new model. And to some degree you need them anyways, to evaluate the models and prompt changes you make.
Overall, as you pointed out, this won't always make sense (either due to the cost of creating examples and training, or simply the lack of available data). But they point this out themselves in the diagram towards the bottom of the page: it only makes sense for frequent and verifiable tasks. This might result in this only being a sensible approach for very large companies (they talk about millions of decisions), but it might make sense for them.
I always wonder how often people in charge of massive compute tasks like LLM pre-training have messed up some detail that invalidates or fails to persist the results and only realise afterwards.
Many of the most valuable companies in the world (who, incidentally, are spending the most money on frontier model inference), already have "the expensive part", the labeled dataset.
The cost of maintaining the datasets and the models is getting commoditized by startups like braintrust and huggingface.
Data drift. The data you are doing real world inference on can start changing, meaning your model performance starts degrading, so the model needs to be retrained on new data (that you have to collect and prepare).
Sure, but that apparently only costs $500. If you do that once a month then that's still basically free. Hell if you do it once a week it's still about as much as a single cheap employee.
you missed the parent comment's point that the $500 training run was merely the last step, generating the training data is likely to be significantly more expensive and time-consuming. might still pay for itself, but it's not a trivial "spend $500 every month" decision.
My company wanted me to help them to fine tune a model. I asked them if I were working full time on it, never heard back.
$500 is largely the cheapest part of it, but I know from experience, most people are clueless about data.
I worked for a billion+ revenue business where they didn't realize one of their key data hasn't been updated for 2 years, until I join them, and these types of screw up is cross business/functions/entities.
What I really started to notice is that SOTA models are really good at putting themselves out of the job.
We can see this already with GPT how luna can do 90% of what sol is used for. The only reason why china still bothers 'distilling' models is accurate training data generation, something that oai and anthropic had to spend years collecting while trying to dodge legal challenges.
The more intelligent models get, the more people will offramp to cheaper solutions that get the job done. There's no real benefit to using a sota model when the accuracy is already 99% and I think that is the biggest danger to US labs.
Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs".
And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc.
But the other fun trick that's been working better and better on the GPT-5.6 series is that even the lower-end models can find the things they didn't think of first when inspecting the already-written output.
I think there's still a bit of hard-to-quantify "creativity" to the bigger models - especially when trying to untangle (a) is this edge case that the model built a complicated way to avoid real/worth worrying about and also (b) even if it is real, is there not a better way to mitigate it? But it might be confirmation bias, in a way that definitely didn't use to be true about GPT-5.3 for planning and Composer 2 for implementation, say.
> even the lower-end models can find the things they didn't think of first when inspecting the already-written output
I use Deepseek V4 Pro for my hobby project (an OpenCode client). The economics will obviously be different at work.
It’s worse than the GPTs, but indeed, if you focus it on reviewing its own code (essentially spending more reasoning and changing perspective), it’s also quite capable at improving its own approaches.
The GPTs are better at general architecting, but I think a lot of the performance gains also came from my more careful prompting (“okay, this is a hard problem, let’s think this through …”) to make using the more expensive model worth it. I use them and they are worth it on subsidized rates, but not at API prices. Since my Codex sub ran out I sometimes miss the models, but it really hasn’t devastated me.
On my own time I still like the 'chat' format of me writing a short prompt and having the model turn it into a few dozen lines of code, or make a simple refactor (like turn a field into a method parameter.
I wasn't impressed with Chinese models (I tried Deepseek V4 Pro/Flash, and GLM5.2) the amount of questionable code and mistakes and misunderstandings meant the money saved and faster speed of models didn't translate to faster progress for me.
That is not overengineering, it’s creating job security:
> Funny story no one will believe, but it’s true. A good friend of mine joined a startup as CTO 10 years ago, high growth phase, maybe 200 devs… In his first week he discovered the company had a microservice for generating new UUIDs. One endpoint with its own dedicated team of 3 engineers …including a database guy (the plot thickens). Other teams were instructed to call this service every time they needed a new ‘safe’ UUID. My pal asked wtf. It turned out this service had its own DB to store every previously issued UUID. Requests were handled as follows: it would generate a UUID, then ‘validate’ it by checking its own database to ensure the newly generated UUID didn’t match any previously generated UUIDs, then insert it, then return it to the client. Peace of mind I guess. The team had its own kanban board and sprints.
> They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...
That’s not overengineering, it’s plain nonsensical, because presumably it doesn’t compare it to all IDs generated in the past. Which, if you wanted to do that, you’d use a database with a uniqueness constraint, in case you don’t already have that anyway.
Fixing this lack of reliable common-sense awareness seems to remain elusive for LLMs.
The ids were used in a limited context and it compared to all the other ids generated in the same context. Yes, absolutely nonsensical.
I find it interesting that presumably the LLM was trained on millions of legit examples where people generate uuids, but still it decided to do something different and unexpected, because it "reasoned" itself into thinking it was necessary.
You can have sol write code and terra will find a plethora of bugs and rip the code apart. In reality this is just the nature of advisory prompting and why advisor from omp.sh is such a great feature. They get caught as they're being written.
Or even more poignant, have terra write code and it will find lots of bugs in its own code. But let it iterate code reviews/fixes/test cases a few times, and you’ll have something nice.
The only cases where weaker models fail entirely and the frontier models really come through is when you have a non-obvious bug in a larger codebase - one that requires tracing lot of calls through the ast (especially in multithreaded code) to understand what might be going wrong.
This is a general theme with technology and the 'S-curve'. Let's not even get into whether the improvement for AI reasoning ability has slowed - for practical purposes of writing a React frontend, it has.
But other tech is like that - I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.
Technology has a tendency to replace new technology, or intrude into vacant areas, but its very rare for technology to replace non-technology (like human interaction).
Most of the recreation humans do in front of screens is tending to (para)social relationships.
I can’t see the difference between terra and sol but I always use Sol. You guys can tell the difference. Even medium to xhigh is not that clear the difference.
I didn't read the Ramp article but this reads like a post hoc fallacy. Companies with 2x revenue have money to spend on AI. Companies with 1.15x revenue don't.
I have great interest in fine-tuning open models, and I'm looking for resources that HN folks can personally recommend. This article looks good and I've bookmarked it to read more thoroughly over time.
I've gotten as far as running Nemotron-3-Nano 30b locally, and plan to target models around 30b - 120b parameters. Based on brief examination of the results I can get, I think these vanilla models are capable enough to add real value, but training could push them over the finish line for specialized tasks.
What I really appreciate is that the author is thinking about the whole process, which is also my goal. Confirmation that others are identifying the same use case, and the same strategy for adding value using this technology.
This is a long term project, so I plan to buy hardware to conduct the fine-tune. ..
What kind of hardware are you using and what is your token generation speed (tok/s)?
Every time I've tried to run local models, even on my beefy mac book (128Gb Ram), I've been very disappointed at tok/s speed.
I've been using dual NVIDIA GPU's with Ollama. Even when I push to 30b models and more context window, Ollama manages RAM / VRAM very well. I never got OOM errors, just massive slowdown as the PCI-E bottleneck throttles the GPU's. I've had some large prompts take 30 minutes to process.
But I should mention: I am trying to implement workflows and processes akin to CI/CD that run 24/7 in the background. These are not interactive use cases, so I don't care so much about tokens per second.
I like the 2x2 grid that describes when to fine-tune a model, when to use a frontier model, etc.
From the article it’s not clear how the scorer grades every episode - was it a frontier model that assigned the grade? How does that continue to work as the model that is being fine-tuned becomes better at the task than the frontier model?
I want small models to win and I am constantly experimenting with them. I have never tried fine-tuning and do not have that kind of budget. My approach is to remove some of the burden from models and bring into the agent.
Tool calling is an example - in some tasks RAG works really well, including coding agents where code, git log, Epics/Tasks, dependencies sources, etc. are all available in very structured manner. You can save many extra tool calls if you can run separate prompts and retrieve the source data needed for the actual work - rather its prompt.
And I really want to focus on search - this is the key technology if we want to use RAG instead of fine-tuning. If we can present really contextual sources in the prompts using a hybrid search approach - you can see how easily we get better results - either decisions or summaries from even small models.
fine tuning a small LLMs or even real small language models (like bert) is what the recommended way since the introduction of LLM. the benefit is pretty much obvious: faster to run, fully controlling the stack, better fit for the custom domain...
but in practice not many people do the fine tuning, for pretty much a single reason: large language models API cost are still very cheap, fast enough, and get improvement all the times. what the point of spend time (and money) to fine tune a specified model, then just when you release it a newer gen generic model is released and beat it?
but if someday the progress for LLM is slowed, or the price increased to the point calling api is not a viable approach any more, then surely the day of fine tuning and small models will come again.
I think we're going to get to that point on the 'S' curve. I also think the moat for a lot of companies is going to be their process and data, self hosting tuned models could be increasingly viewed as a trade secret.
well usually you can just generate the data using LLM. use 2 or 3 different frontier models from different providers, then compare the results and pick the consensus. yes it is not 100% correct like when you make it manually, but then again even human makes mistakes.
This continuous cycle of fine-tuned open models beating frontier on (often vaguely labeled/defined) benchmarks doesn't provide an accurate comparison to the expanding generalized capabilities of the SoTA, which makes them effectively meaningless.
If we were to take these at face value, why is it that the frontier labs' models are making legitimate new discoveries (e.g. Erdős and Jacobian conjectures) and these models are not?
To me, a better signal of capability would be similarly performing novel work at the same or better level, which they presently are not. I say this as someone who very much looks forward to open models being more capable, but to deny the gap is misguided hopeful hype.
Why? Why is the premise that Fine-tuned models should be geared towards new discoveries?
The point of Fine-tuning small models is for specific downstream tasks, which SOTA models can do, but at higher costs. It is purely an economic play, not an attempt at pushing boundaries of SOTA.
I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
If the appropriate qualifiers were added to say "cheap model X does better at test Y than expensive model Z when we fine tune X to take Y test of existing knowledge" then it would be a more accurate statement, but naturally less impressive.
> I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
What if you have to do the task a billion times? Which model will you choose?
This is about saving money by using the right tool for the job. If you have a system that does a lot of mundane, repetitive work, you don't need a frontier model to do it.
It doesn't mean frontier models aren't good at harder tasks.
Huge SOTA models are like a floodlight. They elucidate a huge area at once.
A fine-tuned small model is like a laser pointer. It only illuminates a tiny specific spot. But it can illuminate it as brightly as the huge floodlight, for a tiny fraction of cost.
Depends on use case. That email classifier for legal emails: cheaper at scale as a small tuned model. Let alone better for the planet. Frontier model may have done that tuning!
Amazing! I've been in a similar autoresearch-y rabbit-hole lately with getting Apple's 3B Foundation Model to match Sonnet 4.6 on a very specific task.
The result was: 90% parity achieved with a weird combination of a fine-tuned adapter + 1 deterministic step.
The main issue is the amount of information I have to go through to understand what I'm looking at. It's one of the side effects of writing with AI. Because it sees all tokens at once, it doesn't realize a reader might have no clue at all. What I recommend is trying to reduce the entire thing in 1-2 paragraphs that you wrote yourself. Cleaning up unnecessary prose and tucking away data that not everyone might be immediately interested in.
Share of the maximum achievable score our GRPO-trained 9B open-source model reached on catalog review, vs 76.9% for the best frontier configuration: a 13.5% relative improvement over the frontier, and 36% over its own untrained base (64.2%). The five frontier models, even with optimized prompts, plateaued within a tenth of a point of each other; the trained specialist cleared that ceiling."
_______
This is hard for me to believe. I have a lot of skepticism that frontier models like GPT 5.5 that are likely 2T+ parameters in size only got about 12% more accurate than an untrained 9b parameter LLM.
Why? This is a very narrow task, it’d be surprising if the results were different actually; more interesting question would be how an even smaller model performs in the same finetune.
How difficult is it to post train a model like this. I'm still imagining it being really complicated for someone that has no expert knowledge in this field. Or are there already specialized materials or offers for this that make it easy?
it helps when you first start by distilling (full distribution) domain specific workflows from a larger open model (kimi k3 would be perfect for this now, we used glm 5.2) and then apply your own private RL pipeline to the result. we did this to get a private 27B model for English -> SQL and the results were better than Fable, it lost the ability to explain however.
beating frontier models on highly specialized domains with a tiny fraction of the cost is not difficult to achieve, even when you don't have your own RL pipeline such distillation can still have major savings (27B model will become almost as good as a 3T model on the task). it probably won't work if the task is so ambitious as to not be suited for a specialist.
Ok but a task that works fine on qwen 397b can be finetuned on qwen9b. But in every case so far when building the eval for evaluating the traces I’ve discovered a better prompt that closes the gap better than the finetuning.
They built their own benchmark and then trained directly against its scoring function.. seems to be the rage, but nothing convincing from the article alone.
WTF is happening here? It's a clear marketing piece with clear bunch of bot commenting like the article is a real deal. Is this a regular the modern day HN experience?
The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages. Most use cases are defined within constraints where costs matter a lot.
As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles. As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.
This is what makes open weight models so threatening to them. The political and “it’s China” angle is mostly just a cover for the real reasons why they’re freaked out.
The fact that models are now a pure commodity is bad enough for the big labs. If small open weight models become the norm the big labs are toast.
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages.
Most generic use cases do benefit from a model that has been trained broadly. When you don’t know the specific use case ahead of time, you have to have world knowledge ready to go. Even when coding it’s helpful to have all that knowledge on tap so the model can understand product intent and use cases for the product you’re building.
> As open weight models and cheap fine tuning services become the norm the whole economic framework of these mega models the labs are in an arms race building just completely crumbles
The real expensive part of fine tuning is gathering a good data set. That’s been the hard part of training anything for a long time. If you’re lucky enough to have a neatly organized and clean data set then you can attempt it, but you need to be in a position to run evals and measure quality.
I’ve done it, but there are so many use cases where the engineering, data labeling, and ongoing quality review hours cost so much that it would be cheaper to continue using a frontier lab model that just works from the start.
Can't you take a broadly trained model and throw away useless knowledge (like foreign languages, and 99% of wikipedia) and still end up with a model that is as intelligent and useful as the original but much smaller?
Models are not a collection of training data. The entire model is shaped by the training data it sees during training.
What we call knowledge is encoded as weights. Something we identify as knowledge is the sum of many weights across the model. However the same weight may be involved in very different topics. You can’t point to every weight and say that this one is only for a part of Wikipedia or Harry Potter that I don’t need. That weight could be active for a multitude of topics.
There have been a lot of attempts to reduce the size of mixture of experts models by selectively removing experts. The results are not good, but knowledge isn’t neatly separated within the model.
English reasoning is far better when the model has been trained on many other languages even if you only work in English and never touch another language.
I'm not an AI researcher (would love to hear an actually informed answer on this!), but AFAIK the weights are more or less a black box - you can't easily pinpoint that weights x, y and z relate to Ancient Rome and can therefore be safely removed from a model that's optimised for coding.
You can, and there are many techniques to do so - it's not super precise but you can essentially "delete" concepts out of a model.
I suspect it's a bit like tree-shaking/dead-code elimination in an untyped, dynamic language. You can probably delete a lot of things, but each deletion has a chance to break (or silently degrade) some functionality. Good coding practices, good docs and comments, and good tests can help determine whether a given definition is safe to delete or quickly verify that the deletion was a mistake after the fact. LLMs' weights lack all of those. It's like trying to perform DCE on a binary linked from outputs of ten different compilers with different optimization levels (and the linker is outright bonkers) without running the binary once. Oh, and the binary is many tens of GB in size. Yikes, I'm getting goosebumps just imagining it!
That is not how LLMs work. It isn't some intelligence that is separate from and also contains a database of knowledge that you can prune at will.
if you have the model weights or a provider that will give you the full token distributions you can have the next best thing and do a domain-specific distillation. this way you can take a 9B or 27B LLM and uplift it to the level of the larger model.
it still takes some work and ingenuity, you need to design a workflow that would let you efficiently probe the larger model with your domain-specific prompts, this usually involves a workflow where you obtain the inverse mapping from frontier models (don't need distributions for this step).
Furthermore, shifting the complexity to the model simplifies engineering. A lot of companies made this trade, replacing their bespoke ML models with foundation models.
Very true. In fact, there's a lot more under the surface that will come to light once more efficient tiny models come out.
P.S. Building something that proves that you don't need those many params even.
https://github.com/guilt/tinytot
The premise of your project seems compelling. Is it novel or building on existing work? Anything one could read or watch to get introduced to that area of research?
You would already be able to see RSI and loops being a thing.
So there's going to be more and more research coming out telling you what size of models are really needed to solve a specific task. When you combine that with RSI, the next frontier would be to get this model as close to the real world with instant data and instant reasoning, no fine tuning at all. And see how this one could iterate itself.
I'll also be trying out more tiny models and seeing if they can call each other, and be as effective; You will also be seeing me trying this, in addition to what I'd already mentioned in the other reply.
I did some research about scaling laws and what's unsolved there.
This project was built on structural sparsity, do away with any requirements to train models on largely outdated corpus of data, and only train it on reasoning logic. Because if models can use tools well and retrieve everything else, it greatly would reduce the need for large language models.
Further, when generating traces, I ran a benchmark that showed that on grounded data, n-grams and simple retrieval outperformed transformers on efficiency (See https://github.com/guilt/Transformers) and wanted to try scaling this up to much larger traces.
It worked quite well. Next improvement would be to run this with a BERT model and add more do-not-know traces and the Claude style neurotic self-doubt/self-verify traces and test it out a bit more.
If the entropy required to route people to the relevant answers is very low, full dense attention is a waste of compute. That's the theory being pressure tested here. The more I do this exercise, the more I am getting convinced that smaller models are good enough.
I took a look at the codebase and found that this is just pattern matching. There is no any novelty here. It looks good for knowledge-base retrieval system, but other than that, this is just nothing more than a pattern-matcher.
Would love to be proven wrong with future updates.
Sure. Like throwing money at problems is novel.
Why don't you go and actually build something novel? People need to understand better how current models work, and if retrieval indeed solves 99% of the problems people care about, then those other models aren't doing anything super novel either.
I'm not arguing here: The code is out there, data is out there. Do better if you know something actually better. Thanks!
This is basically the entire argument by Richard Sutton happening in real time:
https://www.youtube.com/watch?v=21EYKqUsPfg
Right now, we keep creating new models with larger and larger weights and throwing hardware at it. His argument is that this is a dead end and ultimately we'll eventually go back to purpose fit algorithms like we've always done in the history of AI development.
Did you even watch the video? Purpose fit algorithms outperform is quite literally the OPPOSITE of Suttons argument. This is the man who wrote this:
“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”
His problem with LLMs are they aren’t general enough… In fact his essay the bitter lesson was what inspired the current LLM scaling.
If you watched the video, you can even see the part where he talks about how AI will fully replace humans as the next stage of life. How wholesome!
It's kind of amazing how badly you've misrepresented Richard Sutton's position. This is the man who coined the term https://en.wikipedia.org/wiki/Bitter_lesson.
The goal is still the singularity though right?
Sure, for business, personal, etc it makes sense to build purpose built but that’s not what these companies are trying to achieve in the end. They want to own a nuclear weapon that outstrips everything else as soon as it exists so they “win”
As someone who worked at multiple startups, I am pretty sure they get it but it doesn’t fit their goals.
They want to moat where they don’t need to compete with other companies because they have something that other companies can’t have.
In my opinion this is a short-sighted and greedy worldview. Haven’t seen it work personally. It is a different version of the month-to-month salary guy thinking he will be a billionaire and having that thrash “mindset”.
The reality is that practically none of those companies will amount to anything and they would be better off weighing the usefulness aspect of their output more. Instead they are imagining they will be Google.
Anthropic and openai ofc are the pinnacle of this greed culture and they correspond to FTX from the crypto trash hype so I don’t think they fit into the scale of sensibility.
Coming from this perspective, it is pretty easy to see what they are.
Going by the chart in the article, if your total workload is 1k cataloged items and your quality threshold is 70%, why wouldn't you just pay $19 to gemini API instead of $500 + time to make a custom fine tune?
The resultant model much cheaper per call and outperforming frontier models. So you'd quickly make your money back at any sort of meaningful scale, continue to pay less from that point on, and have better results. And I think removing a third party dependency is also a benefit in and of itself.
But then can't the case be made that narrower and stricter-defined use cases are better served by more conventional ML? If/Wherever efficiency is a concern, that is.
That's assuming conventional ML and fine-tuned SLMs are interchangeable with acceptable behavioral change/degradation for any specific case
LLMs fit between well defined and poorly defined data; if you have well defined data, you can just feed it to ML models and they'll do what you want.
LLMs will take poorly defined data and "potentially" create well defined output.
So ideal systems will likely be constructed with a LLM on one end and a ML on the other and some how, if you can feedback poor data from the ML back into the LLM to clean up the data, you have a magic layer that doesn't care as much about the structure of the data.
That's a lot of supposition, but that's the difference I see between what we can do with LLMs and how ML models are structured. They might be classified as doing the same thing, but their data inputs are vastly different.
[flagged]
Yeah and these latest and greatest models universally suck hard for any use case outside of the few 'blessed' ones. Like I'm sure their ability to write prose and generally sound like a human being has regressed quite a bit, but even if not. Opus 5 is barely above GPT4 when it comes to stuff like home improvement advice.
Fine timing takes time and data, though. If smart enough models get cheap enough, then most people have lots of use cases that are cost insensitive enough that it's not worth the effort.
Of course, "smart enough" is a low enough threshold for most uses that this is still a problem for the frontier labs.
But at the same time, a truly smart enough closed model could also potentially command almost whatever they'd care to charge for it.
Whether they can actually get to that level remains to be seen, but I can definitely see a situation where most people are perfectly happy with cheap middle of the tree models while large corporations pay magnitudes more than current API pricing for access to models never even marketed as a mass market product and keep the labs afloat.
It's of course be a lot easier for them to find the path towards that of they didn't need to compete with open models in the meantime.
Yes. Fine tuning is an optimization. Like all optimizations, it’s downstream of figuring out your problem, implementing a naive solution, scaling the naive solution, and getting frustrated by SWAP-C. THEN you start poking at what optimizations are possible, choosing an approach, implementing the optimization, and redeploying.
This cycle happens when you have a well defined use case with high volume. It is the far opposite end of the spectrum from the general purpose intelligence on tap that frontier AI models purport to deliver.
I think focused fine tunes and big general models will coexist. Ideally with smart routing and caching to use small, specialized and local options when appropriate.
I mostly agree, but at the same time I wouldn't blindly ignore the "it's China" angle at all. These are models you cannot open up and do a thorough audit of. I do think local compute will eventually be good enough that most people will just use local models. Microsoft might be the ultimate winner of that if they can stop ruining Windows and focus on making an OS similar to how straightforward and simple macOS is, no ads literally everywhere for Microsoft Office. Just make a good OS Microsoft, is that so much to ask? You HAD a really solid OS and you ruined it.
>As does the economic picture that justified the massive infrastructure building that’s now broadly funded by a complex network of debt.
I have a genuine question and I'd like to hear people's good faith thoughts on this.
There's a fair case that open models are threatening to institutions who spent a lot on training proprietary SOTA models.
But to my understanding, the massive investment spend (much of it debt backed as you note) is on data centers, chips, physical infra.
Yes, this infra is needed to train the models, but it is also needed to serve inference.
Perhaps the costs associated with training SOTA models is ultimately a "bust" given open models eroding the SOTA closed model performance advantage.
But demand for inference is skyrocketing and there seems to be no end in sight.
The physical hardware underpinning inference is in fact a scarce good (currently, and this seems sustainable at least over mid-term). And inference is a scarce service as such.
I know that cost of inference constantly goes down as models, technical infrastructure, and applied AI techniques become more efficient (specialized SLMs etc). So this puts downward pressure on prices.
But still... demand for inference is just growing like crazy regardless. Putting upwards pressure on prices.
Doesn't this mean that all the spending on AI infra is much better positioned to get positive ROI regardless of the type of model being served?
Put another way, models seem to be commoditizing, but physical hardware is not (currently).
The vast majority of the AI boom spend is on hardware to my understanding (even training capex can be repurposed for inference).
Doesn't this suggest that the economics for the "railroads level of build-out spending" are healthier than they might seem at first glance?
Maybe. But you have to consider any difference in expected returns. Also account for expected efficiency and capability advances of new hardware. And perhaps most importantly, as of a few years ago the failure rate of these GPUs when run hard was in the ballpark of 10% per year (that might or might not change but regardless it definitely needs to be accounted for).
It seems exceedingly unlikely that we will see growth sufficient to justify the level of investment. But as you say that doesn't necessarily mean a worst case scenario either. An investment can be bad without being ruinous after all.
seems that, like software engineering and other areas before, they also have to rediscover that one single monolithic solution that handles everything is too inflexible and not maintainable, it's just a bad approach. people don't need the models they use to generate their codebase to also be able to translate Shakespeare into gen-z slang
Im pretty sure they know quite well but its not their current focus on creating or allowing others to create small finetuned and optimized models.
In the race they are in, its still highly beneficial to be the frontier model.
I use the frontier model every single day through my company and my company happily spends these tokens.
It makes a huge difference if different people can use one interface to do everything.
Big models give you fundamental things: A lot of facts/context, usability (you might be a native english speaker, don't underestimate how hard it is for A LOT of people to formulate what they want/need in english only AND a low complexity.
No one needs to build a router and x sub models and a router architecture. You literaly just have an API, you might choose the model and the effort but thats it.
I find this current state of the art a LOT more telling on the current progress we are in than anything else. I'm confident that small optimized models will become a lot more relevant like lets say java + english + a second language + spring boot + postgresql. It might also be beneficial in the long term to finetune with your project details.
For a lot of very technical non human interfacing things, finetuning is happening left and right.
But what i find very interesting is emerging complexity capabillity. I believe that fables skill to hold more topics and combine more complex solutions together is because of its parameter size.
We will have to figure out if we can extract this complexity out of it while reducing the training data in a way that the training data focus more on thinking. Plenty of smaller thinking models show that this is doable.
Btw. Mixture of Experts is for sure not optimal for this, but it already is a form of optimized sub models. Perhaps we might just have MoE with a million experts in the future. One per lanuage + area of expertise etc.
Also don't forget: IF AGI is coming through a current frontier model, you will let it work for hours, days and weeks on one problem completly independent of any human input and it will be better than a human. If they reach this before a collapse, we are done and they 'won'. For this you need big frontier models.
I’m not sure it will go this direction. Using a generic (near) frontier model is often cheap enough that you need to talk really big volumes before it pays off to fine tune.
My example: we were doing single digit millions of automated call summaries a few years back at a major bank with GPT-4o. Smaller model gave more rejected summaries (compliance not happy), so we briefly looked at fine tuning a smaller model and basically concluded that even at that scale the effort of data collection, management, fine tuning, hosting the model, etc didn’t have a sufficient business case vs picking up other projects.
I actually prefer using less powerful models most of the time, I might use a stronger model initially and then switch to weaker models as I fine tune the output.
Offloading too much of your task to models eliminates the human ownership, without ownership you can't move forward. Context sizes can't keep up with large codebases and markdown files with instructions and guidelines only take the LLM so far in the ownership aspect.
My gut feeling is that to start offloading ownership to the LLM we would need to see at least two order of magnitude increases on context size.
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages.
I think that's true across the board. Along those lines, most companies all of a sudden started looking for AI researchers, instead of software engineers, which is what they still need :-).
Just like a decade-ish ago, when the FAANG interview style got openly known, then everybody started to conduct interviews in that pattern :-)
what are those cheap fine-tuning services these days?
Tinker is pretty cheap. Prime Intellect if you want more flexibility.
unsloth for small use cases, tinker for enterprise level use cases?
I haven’t used their platform so can’t speak to this, but super brilliant team and big fans of their work overall!
I would not say cheap per say, but in the context of "where is the business success going to be" that's one of the reasons why Mistral focuses on providing tuned model on premises to their customers.
Some specialized models may end solving themselves by helping fit the problem with the appropriate regular algorithms/formulas, which are a million times more efficient. So they are probably less attractive to dump money on. As of currently they still benefit from expressing lots of patterns that nobody bothered formalizing.
There is no such thing as extracting "task specific intelligence" into a cheaper model.
News to me. What do you call mixture-of-experts and model distillation?
those are still general purpose models. do you have an example of task specific model?
> The point that the major labs don’t seem to get is that the vast majority of use cases simply don’t need models that have 50 PhDs and can speak 12 languages
I think they understand it but they think they can get away with it because they own the narrative. As long as they can make people believe they need such a model, it doesn't matter if it's true or not.
They are playing the cloud playbook, it didn't matter that most companies didn't need 99,999% uptime and instantaneous horizontal scaling, as long as people believed they did they are happily paying 10-100x the cost to AWS instead.
the major labs want to advance science. current business use cases are a happy accident
That party is over. They’re all on the clock now to show they can make money or the plug will be pulled.
Every time I see this kind of story, two things bother me.
First, I have watched the free improvement of frontier models surpass the gains from retraining, many times now. Squeezing more out of the models that already exist, or simply doing nothing and waiting, is a real strategy and it often pays better. The fair comparison is not against today's frontier but against whatever ships while you are still maintaining your fine-tune.
Second, the $500 training bill is the cheapest line item in this story. The expensive parts are creating the data and maintaining the model afterwards. How many use cases can actually produce 177k scored episodes? Here they had to generate them synthetically from Amazon Berkeley Objects. To me, that dataset is the strongest evidence in the article of how hard fine-tuning is to apply: if the data existed naturally, nobody would need to manufacture it.
Regarding your first point, if the cost of not doing it while you wait exceeds the cost of training/maintaining the model, then this approach still makes sense. This really depends on how fast you expect cheap (and open-source) models to improve. So it might only be a temporary strategy, but still worthwhile. Also, I expect that specialized models will always be better (cheaper or better outcomes) than a general model, similar to how specialized HW like GPUs is still used even though CPUs have improved a lot too.
Maintaining in the way of ongoing training is relatively cheap, considering that the initial training run is only 500$ (in this example). Creating new examples to account for drift in the training data is more expensive, but these examples can then be reused when training a new model. And to some degree you need them anyways, to evaluate the models and prompt changes you make.
Overall, as you pointed out, this won't always make sense (either due to the cost of creating examples and training, or simply the lack of available data). But they point this out themselves in the diagram towards the bottom of the page: it only makes sense for frequent and verifiable tasks. This might result in this only being a sensible approach for very large companies (they talk about millions of decisions), but it might make sense for them.
And the hidden, trial and error runs tweaking the hyperparams. guaranteed, it cost >$5000 in training runs alone.
I always wonder how often people in charge of massive compute tasks like LLM pre-training have messed up some detail that invalidates or fails to persist the results and only realise afterwards.
Not to mention the labor cost.
Nope, that was the total cost
On your second part:
Many of the most valuable companies in the world (who, incidentally, are spending the most money on frontier model inference), already have "the expensive part", the labeled dataset.
The cost of maintaining the datasets and the models is getting commoditized by startups like braintrust and huggingface.
What do you mean with maintaining the model? I am puzzled.
Data drift. The data you are doing real world inference on can start changing, meaning your model performance starts degrading, so the model needs to be retrained on new data (that you have to collect and prepare).
Sure, but that apparently only costs $500. If you do that once a month then that's still basically free. Hell if you do it once a week it's still about as much as a single cheap employee.
you missed the parent comment's point that the $500 training run was merely the last step, generating the training data is likely to be significantly more expensive and time-consuming. might still pay for itself, but it's not a trivial "spend $500 every month" decision.
My company wanted me to help them to fine tune a model. I asked them if I were working full time on it, never heard back. $500 is largely the cheapest part of it, but I know from experience, most people are clueless about data.
I worked for a billion+ revenue business where they didn't realize one of their key data hasn't been updated for 2 years, until I join them, and these types of screw up is cross business/functions/entities.
What I really started to notice is that SOTA models are really good at putting themselves out of the job.
We can see this already with GPT how luna can do 90% of what sol is used for. The only reason why china still bothers 'distilling' models is accurate training data generation, something that oai and anthropic had to spend years collecting while trying to dodge legal challenges.
The more intelligent models get, the more people will offramp to cheaper solutions that get the job done. There's no real benefit to using a sota model when the accuracy is already 99% and I think that is the biggest danger to US labs.
The upper end is all about coding. If I have terra on extra high write code, Sol will find a plethora of bugs and rip the code apart.
Anything else? Sure use a cheaper model.
Sometimes that'll turn up real bugs, sometimes just overengineered designs, premature-optimization, and 1-in-a-million possibility "bugs".
And sometimes it's not about the model, it's just about refining the search space. E.g. I've had Opus write tests and GPT 5.5 write the implementation passing all the tests. Then ask about that specific implementation and find some real corner cases. Add those to tests, etc.
But the other fun trick that's been working better and better on the GPT-5.6 series is that even the lower-end models can find the things they didn't think of first when inspecting the already-written output.
I think there's still a bit of hard-to-quantify "creativity" to the bigger models - especially when trying to untangle (a) is this edge case that the model built a complicated way to avoid real/worth worrying about and also (b) even if it is real, is there not a better way to mitigate it? But it might be confirmation bias, in a way that definitely didn't use to be true about GPT-5.3 for planning and Composer 2 for implementation, say.
> even the lower-end models can find the things they didn't think of first when inspecting the already-written output
I use Deepseek V4 Pro for my hobby project (an OpenCode client). The economics will obviously be different at work.
It’s worse than the GPTs, but indeed, if you focus it on reviewing its own code (essentially spending more reasoning and changing perspective), it’s also quite capable at improving its own approaches.
The GPTs are better at general architecting, but I think a lot of the performance gains also came from my more careful prompting (“okay, this is a hard problem, let’s think this through …”) to make using the more expensive model worth it. I use them and they are worth it on subsidized rates, but not at API prices. Since my Codex sub ran out I sometimes miss the models, but it really hasn’t devastated me.
On my own time I still like the 'chat' format of me writing a short prompt and having the model turn it into a few dozen lines of code, or make a simple refactor (like turn a field into a method parameter.
I wasn't impressed with Chinese models (I tried Deepseek V4 Pro/Flash, and GLM5.2) the amount of questionable code and mistakes and misunderstandings meant the money saved and faster speed of models didn't translate to faster progress for me.
They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...
I wonder if that's also how I'd behave if I had a reinforcement learning harness around me that dived hard on and punished me for every small mistake.
That is not overengineering, it’s creating job security:
> Funny story no one will believe, but it’s true. A good friend of mine joined a startup as CTO 10 years ago, high growth phase, maybe 200 devs… In his first week he discovered the company had a microservice for generating new UUIDs. One endpoint with its own dedicated team of 3 engineers …including a database guy (the plot thickens). Other teams were instructed to call this service every time they needed a new ‘safe’ UUID. My pal asked wtf. It turned out this service had its own DB to store every previously issued UUID. Requests were handled as follows: it would generate a UUID, then ‘validate’ it by checking its own database to ensure the newly generated UUID didn’t match any previously generated UUIDs, then insert it, then return it to the client. Peace of mind I guess. The team had its own kanban board and sprints.
https://news.ycombinator.com/item?id=48061235
> They do tend to overengineer. The other day 5.6 Sol generated a while loop around a uuid4 call to make sure the generated ids were unique...
That’s not overengineering, it’s plain nonsensical, because presumably it doesn’t compare it to all IDs generated in the past. Which, if you wanted to do that, you’d use a database with a uniqueness constraint, in case you don’t already have that anyway.
Fixing this lack of reliable common-sense awareness seems to remain elusive for LLMs.
The ids were used in a limited context and it compared to all the other ids generated in the same context. Yes, absolutely nonsensical.
I find it interesting that presumably the LLM was trained on millions of legit examples where people generate uuids, but still it decided to do something different and unexpected, because it "reasoned" itself into thinking it was necessary.
You can have sol write code and terra will find a plethora of bugs and rip the code apart. In reality this is just the nature of advisory prompting and why advisor from omp.sh is such a great feature. They get caught as they're being written.
Or even more poignant, have terra write code and it will find lots of bugs in its own code. But let it iterate code reviews/fixes/test cases a few times, and you’ll have something nice.
The only cases where weaker models fail entirely and the frontier models really come through is when you have a non-obvious bug in a larger codebase - one that requires tracing lot of calls through the ast (especially in multithreaded code) to understand what might be going wrong.
This is a general theme with technology and the 'S-curve'. Let's not even get into whether the improvement for AI reasoning ability has slowed - for practical purposes of writing a React frontend, it has.
But other tech is like that - I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.
Technology has a tendency to replace new technology, or intrude into vacant areas, but its very rare for technology to replace non-technology (like human interaction).
Most of the recreation humans do in front of screens is tending to (para)social relationships.
> I don't even remember when I bought my LCD TV - 2018 I think? I have no inclination of buying a new one.
LCD backlights are usually rated for 5-10 years of normal use so your inclination might change soon.
Eh, I’m using an 15-year old LCD monitor and its still fine. One generally doesn’t use these at full brightness.
I can’t see the difference between terra and sol but I always use Sol. You guys can tell the difference. Even medium to xhigh is not that clear the difference.
Anything but Sol is not sufficient for complex (or even just large) enough code.
I didn't read the Ramp article but this reads like a post hoc fallacy. Companies with 2x revenue have money to spend on AI. Companies with 1.15x revenue don't.
Alternatively, AI-heavy companies are much more likely to be more agile growth-stage companies rather than large mature companies.
I have great interest in fine-tuning open models, and I'm looking for resources that HN folks can personally recommend. This article looks good and I've bookmarked it to read more thoroughly over time.
I've gotten as far as running Nemotron-3-Nano 30b locally, and plan to target models around 30b - 120b parameters. Based on brief examination of the results I can get, I think these vanilla models are capable enough to add real value, but training could push them over the finish line for specialized tasks.
What I really appreciate is that the author is thinking about the whole process, which is also my goal. Confirmation that others are identifying the same use case, and the same strategy for adding value using this technology.
This is a long term project, so I plan to buy hardware to conduct the fine-tune. ..
What kind of hardware are you using and what is your token generation speed (tok/s)? Every time I've tried to run local models, even on my beefy mac book (128Gb Ram), I've been very disappointed at tok/s speed.
I've been using dual NVIDIA GPU's with Ollama. Even when I push to 30b models and more context window, Ollama manages RAM / VRAM very well. I never got OOM errors, just massive slowdown as the PCI-E bottleneck throttles the GPU's. I've had some large prompts take 30 minutes to process.
But I should mention: I am trying to implement workflows and processes akin to CI/CD that run 24/7 in the background. These are not interactive use cases, so I don't care so much about tokens per second.
I like the 2x2 grid that describes when to fine-tune a model, when to use a frontier model, etc.
From the article it’s not clear how the scorer grades every episode - was it a frontier model that assigned the grade? How does that continue to work as the model that is being fine-tuned becomes better at the task than the frontier model?
turns out the bottleneck was never model size, it was having someone who actually understood the problem define the reward function
I agree very much. I'm getting the same take-away even more now that I'm using autoresearch as my main strategy.
And I wonder: We've got all these amazing (programming) languages to define solutions; where are the languages to clearly define the problems?
I want small models to win and I am constantly experimenting with them. I have never tried fine-tuning and do not have that kind of budget. My approach is to remove some of the burden from models and bring into the agent.
Tool calling is an example - in some tasks RAG works really well, including coding agents where code, git log, Epics/Tasks, dependencies sources, etc. are all available in very structured manner. You can save many extra tool calls if you can run separate prompts and retrieve the source data needed for the actual work - rather its prompt.
And I really want to focus on search - this is the key technology if we want to use RAG instead of fine-tuning. If we can present really contextual sources in the prompts using a hybrid search approach - you can see how easily we get better results - either decisions or summaries from even small models.
fine tuning a small LLMs or even real small language models (like bert) is what the recommended way since the introduction of LLM. the benefit is pretty much obvious: faster to run, fully controlling the stack, better fit for the custom domain...
but in practice not many people do the fine tuning, for pretty much a single reason: large language models API cost are still very cheap, fast enough, and get improvement all the times. what the point of spend time (and money) to fine tune a specified model, then just when you release it a newer gen generic model is released and beat it?
but if someday the progress for LLM is slowed, or the price increased to the point calling api is not a viable approach any more, then surely the day of fine tuning and small models will come again.
I think we're going to get to that point on the 'S' curve. I also think the moat for a lot of companies is going to be their process and data, self hosting tuned models could be increasingly viewed as a trade secret.
It's also really hard. And you need a lot of data for the fine tuning to produce good results.
well usually you can just generate the data using LLM. use 2 or 3 different frontier models from different providers, then compare the results and pick the consensus. yes it is not 100% correct like when you make it manually, but then again even human makes mistakes.
This continuous cycle of fine-tuned open models beating frontier on (often vaguely labeled/defined) benchmarks doesn't provide an accurate comparison to the expanding generalized capabilities of the SoTA, which makes them effectively meaningless.
If we were to take these at face value, why is it that the frontier labs' models are making legitimate new discoveries (e.g. Erdős and Jacobian conjectures) and these models are not?
To me, a better signal of capability would be similarly performing novel work at the same or better level, which they presently are not. I say this as someone who very much looks forward to open models being more capable, but to deny the gap is misguided hopeful hype.
Why? Why is the premise that Fine-tuned models should be geared towards new discoveries?
The point of Fine-tuning small models is for specific downstream tasks, which SOTA models can do, but at higher costs. It is purely an economic play, not an attempt at pushing boundaries of SOTA.
I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
If the appropriate qualifiers were added to say "cheap model X does better at test Y than expensive model Z when we fine tune X to take Y test of existing knowledge" then it would be a more accurate statement, but naturally less impressive.
> I'm not suggesting that fine-tuned models don't have their place, all I'm saying is that the constant drumbeat of "cheap model X beats more expensive model Z" completely misses that the more expensive model is capable of doing more things at a higher level.
What if you have to do the task a billion times? Which model will you choose?
Maybe because people who are target audience don’t need to have it spelled out like that?
People who are not really into it, don’t care.
Speed play also you can get much faster responses with 9b model.
This is about saving money by using the right tool for the job. If you have a system that does a lot of mundane, repetitive work, you don't need a frontier model to do it.
It doesn't mean frontier models aren't good at harder tasks.
That's fair and I agree with this framing.
That was always the framing. I don’t understand your pedantry.
Huge SOTA models are like a floodlight. They elucidate a huge area at once.
A fine-tuned small model is like a laser pointer. It only illuminates a tiny specific spot. But it can illuminate it as brightly as the huge floodlight, for a tiny fraction of cost.
Depends on use case. That email classifier for legal emails: cheaper at scale as a small tuned model. Let alone better for the planet. Frontier model may have done that tuning!
Amazing! I've been in a similar autoresearch-y rabbit-hole lately with getting Apple's 3B Foundation Model to match Sonnet 4.6 on a very specific task.
The result was: 90% parity achieved with a weird combination of a fine-tuned adapter + 1 deterministic step.
I wrote the whole thing down here: https://alexisrondeau.me/tada/research/FMDiscovery/dashboard... which includes the question, the answer, the 96 experiments and their lineage etc. etc.
> I wrote the whole thing down here
I like the idea, but it looks really hard to read. Try reading it top to bottom without skipping.
Oh, thank you for noticing.
Is it a layout issue for you (it lays out better on desktop) or the language of the text or...? Let me know!
The main issue is the amount of information I have to go through to understand what I'm looking at. It's one of the side effects of writing with AI. Because it sees all tokens at once, it doesn't realize a reader might have no clue at all. What I recommend is trying to reduce the entire thing in 1-2 paragraphs that you wrote yourself. Cleaning up unnecessary prose and tucking away data that not everyone might be immediately interested in.
A fine-tune will generally outperform just about any other method on a closed domain, non generative problem.
LLMs are great because they can handle open domain problems in part because they are generative.
"87.3%
Share of the maximum achievable score our GRPO-trained 9B open-source model reached on catalog review, vs 76.9% for the best frontier configuration: a 13.5% relative improvement over the frontier, and 36% over its own untrained base (64.2%). The five frontier models, even with optimized prompts, plateaued within a tenth of a point of each other; the trained specialist cleared that ceiling."
_______
This is hard for me to believe. I have a lot of skepticism that frontier models like GPT 5.5 that are likely 2T+ parameters in size only got about 12% more accurate than an untrained 9b parameter LLM.
Why? This is a very narrow task, it’d be surprising if the results were different actually; more interesting question would be how an even smaller model performs in the same finetune.
There seems to be no hold out data for test, so this is just overfitting?
RL is the gold standard and significantly beats self learning methods, but other than coding and computer refereed games it's cost prohibitive.
How difficult is it to post train a model like this. I'm still imagining it being really complicated for someone that has no expert knowledge in this field. Or are there already specialized materials or offers for this that make it easy?
it helps when you first start by distilling (full distribution) domain specific workflows from a larger open model (kimi k3 would be perfect for this now, we used glm 5.2) and then apply your own private RL pipeline to the result. we did this to get a private 27B model for English -> SQL and the results were better than Fable, it lost the ability to explain however.
beating frontier models on highly specialized domains with a tiny fraction of the cost is not difficult to achieve, even when you don't have your own RL pipeline such distillation can still have major savings (27B model will become almost as good as a 3T model on the task). it probably won't work if the task is so ambitious as to not be suited for a specialist.
The problem i've had with finetuning models is that most of the time better prompting beats finetuning
Better prompting doesn't improve response time or price!
Ok but a task that works fine on qwen 397b can be finetuned on qwen9b. But in every case so far when building the eval for evaluating the traces I’ve discovered a better prompt that closes the gap better than the finetuning.
Yes, it can.
[flagged]
What benchmark is it? Is it super niche?
They built their own benchmark and then trained directly against its scoring function.. seems to be the rage, but nothing convincing from the article alone.
https://huggingface.co/papers/2607.23806 this is a cheaper solution
No mention of training/test/holdout dataset splits anywhere in this article - so how am I to know whether or not this is some overfit result?
is this an ad?
| Harvey's legal agent beats GPT-5.5 and Claude Opus 4.8 on its own rubrics, and Intercom's Fin Apex resolves more support issues at lower cost.
The rubric and cost argument here just casually ignores all of the other challenges and real business issues of evolving models over time
A 9B beating frontiers" No! But nailing specialized tasks Yes.
Well I surely hope so. That’s the whole point of fine tuning.
Comparing the revenue of the top quartile of AI-using companies to the average of all non-AI-using companies is beyond intellectually dishonest.
WTF is happening here? It's a clear marketing piece with clear bunch of bot commenting like the article is a real deal. Is this a regular the modern day HN experience?
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
Tldr: we don't know what we are doing like the rest of 99% AI teams.
[dead]