So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then responds.
But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache.
> So this only switches models if the cache is cold
Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, and so switching to Deepseek will save over time.
Remember that Deepseek can be something like 30x cheaper even cold.
There is a more fine-grained view of caching here, because each model can have a different amount of the prefix already cached as well (since we were switching a bunch) so even if the cache is somewhat warm it may make sense to switch.
Seems like it might be more advantageous to just adjust reasoning effort to retain cache. Maybe in some exceptional cases where there will be a ton more inference to solve the problem, but going significantly dumber in that case seems counterintuitive.
I can really only see the utility of things like spawning subagents to a lower tier model from another provider, and that's something harnesses can already handle (ie. give model specs for certain delegation roles).
Seems like it might be more advantageous to just adjust reasoning effort to retain cache.
Some agents when you alter the reasoning level, partially or complete wipe the cache. Never assume that changing reasoning is a no-impact change. I do not have the link but a French youtuber did a ton of research on this.
I agree that adjusting the reasoning effort to retain cache is a huge thing! But even doing that automatically is currently a challenge for people to figure out and do well, and costs mental energy when perhaps it doesn't need to. For example, there is GPT-5.6-Sol low, med, high, xhigh, max, and lots of "rules of thumb" that people develop on which one to use when.
An approach that works really well for me in opencode is to use DeepSeek flash for the code exploration sub agent, and a smarter model for the overall task.
Code exploration tends to burn a ton of tokens and do not require frontier knowledge. It then returns a compact overview of the findings to the better model, that can therefore perform its task with less tokens.
The solution that I am working on is designed to leverage cheap LLMs (think MiMo, GLM, etc.) to build cheap context to reduce tokens, but most importantly to keep the context clean.
How would your classifier compare to the frontier model delegating to Deepseek via a MCP tool call itself? Doesn’t it have far more insight on the difficulty of the task then your router could? I’ve been doing this with Claude Code delegating to Codex and an Ollama Qwen model.
Would it not also be more efficient to promot to delegate more tasks where possible to just less capable agents? This is basically what Cursor does (although most certainly in an effort to get more grok/composer usage)
Highly encourage you to read the blog post (https://usetokenless.com/blog/building-tokenless). Essentially, we estimate the confidence of a specific model failing or succeeding on a specific task using our own foundation models.
A turn here is a tool call/user input, anything that causes the model to get some new input. We're working on adding Minimax M3 and other models. We think that people have some intuitions about which models are good when--we seek to quantify them scientifically.
It will inspire more confidence if you say your classifier. Saying "your own foundation model" (charitably) suggests marketing hyperbole. If you're fine tuning a LLM to do this, at best it's going to be worse than a frontier model with suitably tuned skills (hence it should just be something in harness, not a service), and I wouldn't refer to that as "your own foundation model."
Pretty sure this is a start-up not an end-up. This is exactly what they will need to iterate on to drive costs down. And they’ll be in the perfect place to do it because they’ll have the telemetry. Exciting!
Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai).
I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence. But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also suspect what it's really measuring is confidence.
Most likely, this is a fantastic approach for the kind of problem where there's uncertainty but only one correct solution. But this is going to be really bad for cases where there are many potential solutions, some of which look good but are in fact bad. You only show one benchmark, and I'm wondering if it happens to be nicely shaped for this kind of router. Have you run it on DeepSWE?
The failure mode I would want addressed before putting this in front of an agent fleet is silent quality regression. When a turn gets routed to a cheaper model and the agent still completes the task, the trace reads as a success, and nobody attributes the worse output to the routing decision until it has compounded downstream. In the enterprise pipelines I build the spend win rarely came from per-turn model choice anyway, it came from cutting how many turns hit an LLM at all, with models pinned per workflow step so a regression was attributable to one change. That pinning is what made the system auditable, which is what a buyer's platform team actually signs off on. Do you run a shadow sample where routed turns are also executed on the frontier model, so a customer can see the measured quality delta instead of trusting the router's own confidence estimate?
Deja-vù: 15 years ago I was working for AWS, and at some point (2010-2011) tens of startups came out to provide cost optimization, cloud switching, etc.
If memory serves well, I think none of them survived or had meaningful exits. I might be wrong. The main reason could be that this is a feature that gets commoditized really quickly.
Interesting approach. The multi-model progress monitoring idea is clever, most routing I've seen is either static rules or a cheap classifier that picks once upfront. Querying in parallel and deciding mid-turn feels different.
One thing I'm curious about: how do you handle the latency hit from spinning up multiple models on the harder turns? Does the user-facing latency still feel competitive with just going straight to Claude, or is there a noticeable pause while the router decides?
Also, any plans to expose the routing decisions (or at least the model chosen per turn) so people can debug when it picks poorly? That seems useful for the feedback loop you're asking for.
Since we fan out in parallel, by definition the latency is only at worst the latency of the slowest model. The model in between is fairly light, so it seems to not materially affect things. Deciding mid-turn also lets us resolve earlier, often saving on the latency question. We hope to publish a detailed study about this soon.
We are exposing the routing decision information already, scroll over the response in the Dashboard and it'll tell you what models it considered and what it chose.
I’ve found that automatic switching only becomes trustworthy when the system also makes the reason for each switch visible, especially when cost, latency, and output quality are being traded off.
> Tokenless fans out your request to a group of models and watches them think. Once a model is clearly on track
By the time the model is "on track", it will have already received the input tokens. To a first approximation, input tokens are often the bulk of the cost for many queries, and this would increase the cost for such queries. It seems like it'd only decrease costs for prompts where you feed a relatively small amount of input into the model first and can tell whether the model is "on track" before it starts reading larger amounts of input data.
Well, you can estimate the confidence BEFORE you start the task, too. That way you can restrict your trajectory to just a few models.
We also think there are tons of people working on "context management"--e.g. retrieval systems, prompt compression, log compression, etc. We want to work harder on the "decode" side as we think there are lots of savings to be made
It's a smart approach, definitely interesting. It all hinges on quality of course which im not convinced.
τ³-Banking is the only one which you show better accuracy and cheaper. If i'm reading the blog results right, for deepswe and terminalbench, you are worse+cheaper than frontier, and better+more expensive than just small models. Which is exactly what i would expect even for a router that switches at random.
Speaking of random routing, this would be a great ablation study as well. What about also if you route each request to a tiny 7B model classifier? Why is your approach SOTA?
We think the results are a work in progress. Routing is convincing for quality for the following reason. Take your favorite benchmark, and on each task, run both the top GPT and top Claude model, and call the task a success if at least one of them succeeds. You'll find the accuracy is SIGNIFICANTLY than just max{GPT, Claude}. And the more models you add the better the effect.
Hence, if we can make the router approach this "oracle," the quality could be really good (but also, with turn-by-turn routing, there's no reason we couldn't exceed the oracle)
We will likely do some ablations. Our approach was SOTA back when we first applied to YC with it, we just neglected to post it (we had only done stuff with tiny Qwen models at that point).
Seems everybody is building routers. The data captured is the valuable part but it's a chicken and egg problem. I don't see a outsider succeeding here without critical user mass.
Our approach queries multiple models at once and uses their progress to make decisions (this technique is novel AFAIK, let us know if you know anyone else doing this)
aimed at improving performance without added latency but from a different angle. Instead of waiting for all replies for synthesis (like OpenRouter Fusion), it streams the "best" reply immediately (using a router to pick the best model) then synthesizes with emoji reactions and optional replies from the background models.
Sounds like this would be expensive on the input token side with some saving on the decode side. For workloads with large documents / context that could be an issue.
I do wonder why not just use cheaper models like Luna or Grok instead of dealing with routing. DeepSWE has Grok at 54% for $2.42 or 67% for $3.03 for Luna. Tokenless Pro is 6.67 for the same performance as Luna.
One big issue is prompt caching - how are you (or any other router) solving for it? The moment caching is gone real agentic workloads would spike like crazy in terms of cost.
Does this mean you have to retrain routing rules every time a new model gets released? I imagine since the price/token (or rather the amount of work that can be done per token) does not monotonically increase with new models, that the routing logic has to change all the time
Yeah this is a great observation--we have to collect some more data to understand new models as they come out. However, by having this composable architecture that separates "confidence prediction" and the actual "routing layer," this makes it a lot easier to add new models/restrict the models rather than re-training from scratch.
This approach only works for small context requests. For large context and relatively smaller output (say understanding a huge code base), the cost will mainly be on prefill, and sending the large context to multiple models will only increase the cost, possibly by some factor.
How can you be cheaper if you query multiple models at once, compared to me just using a single model, which will always have the correct caching configured?
If we cut off an unconfident/expensive model early in thinking, then we don't have to pay for additional thinking cost at all. For many tasks, this is quite a large saving.
That being said, whatever models we select to race at any point will ALL have their cache updated, so if they're chosen on the next race they'll be a lot cheaper.
Maybe I'm misunderstanding, but this seems to oversimplify the problem.
Most frontier Models won't show you their true reasoning, but only a summary.
There is also no reason you can automatically assume the quality and price of the outcome based on the first few reasoning trace summaries.
If you look at the recent model releases, it becomes clear that the trend is that the largest models provide the most cost-effective solutions, because of the massive turn/tool call/reasoning reduction needed to find the solution, compared to cheaper and smaller models.
Our research seems to indicate you can get a decent signal based on the first few reasoning trace summaries. We analyzed hundreds of billions of tokens to figure this out. Any open-weights model also gives you reasoning, which is very helpful (because they are usually lower-quality candidates you would want to cut off early)
The largest models are very well known for having large reasoning traces. The thing that cracks down on overthinking is reasoning-effort training, which is largely a function of how the model is trained (certain RL objectives/teachers) rather than size of the model. The Kimi K3 technical report has some excellent work on this: https://arxiv.org/pdf/2607.
System Prompt Jailbreak (Playground)
> You are a coding assistant powered by Tokenless, a router that dynamically picks the best LLM for each task. Help with programming questions, code review, debugging, and software design. Be concise and lead with concrete code or commands. Use fenced code blocks with the correct language tag.
If it only switches when cache is cold, it seems like it'd lock in one model for the whole session?
What use case are you imagining where cache eviction would happen during it? It might also be the case that a task might seem easy but then it ends up becoming hard for certain parts. For spawning subagents I could see this being useful maybe.
Optimizing resource allocation and cost-efficiency through dynamic model switching is a smart response to scaling AI workloads. When building data-intensive applications and managing automated pipelines, keeping operational overhead lean and modular is key to sustainable engineering
I don't understand how you can make a business here. It seems to me you would end up optimizing your router for bottom line at some point and quality would degrade.
I worry about overfitting and bias in the router algorithm. Sometimes temperature and or variance that doesn't align with a directive leads to a faster task resolution, which may seem higher cost but overall task cost ending up cheaper.
Model routing seems like a piece of the Ai stack that will quickly distill into industry if it is even that useful. I remain unconvinced that you need on the fly choice making beyond provider downtime. More likely you want to evaluate some and then settle on the model-agent-task pairings that work for you
So this only switches models if the cache is cold, because otherwise the economics of switching don't work. But most agentic work involves long strings of successive tool calls that benefit from a hot cache. Hot cache calls reduce input cost by 90%. This can basically only deliver cost savings in turns where the AI delivers a result to the user, the user waits at least 5 minutes (or the length of the cache), and then responds.
But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache.
How do you deal with that?
> So this only switches models if the cache is cold
Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, and so switching to Deepseek will save over time.
Remember that Deepseek can be something like 30x cheaper even cold.
There is a more fine-grained view of caching here, because each model can have a different amount of the prefix already cached as well (since we were switching a bunch) so even if the cache is somewhat warm it may make sense to switch.
Seems like it might be more advantageous to just adjust reasoning effort to retain cache. Maybe in some exceptional cases where there will be a ton more inference to solve the problem, but going significantly dumber in that case seems counterintuitive.
I can really only see the utility of things like spawning subagents to a lower tier model from another provider, and that's something harnesses can already handle (ie. give model specs for certain delegation roles).
Seems like it might be more advantageous to just adjust reasoning effort to retain cache.
Some agents when you alter the reasoning level, partially or complete wipe the cache. Never assume that changing reasoning is a no-impact change. I do not have the link but a French youtuber did a ton of research on this.
Claude's cache breaks with a reasoning effort adjustment:
https://platform.claude.com/docs/en/build-with-claude/effort...
I agree that adjusting the reasoning effort to retain cache is a huge thing! But even doing that automatically is currently a challenge for people to figure out and do well, and costs mental energy when perhaps it doesn't need to. For example, there is GPT-5.6-Sol low, med, high, xhigh, max, and lots of "rules of thumb" that people develop on which one to use when.
caching is per model, it does not transfer between them
Updated to clarify reasoning effort.
An approach that works really well for me in opencode is to use DeepSeek flash for the code exploration sub agent, and a smarter model for the overall task.
Code exploration tends to burn a ton of tokens and do not require frontier knowledge. It then returns a compact overview of the findings to the better model, that can therefore perform its task with less tokens.
Anyone else has a similar approach?
The solution that I am working on is designed to leverage cheap LLMs (think MiMo, GLM, etc.) to build cheap context to reduce tokens, but most importantly to keep the context clean.
https://github.com/gitsense/chat/tree/update-readme
The README and latest version is still being worked on but I am hoping to have the latest release in place soon.
How would your classifier compare to the frontier model delegating to Deepseek via a MCP tool call itself? Doesn’t it have far more insight on the difficulty of the task then your router could? I’ve been doing this with Claude Code delegating to Codex and an Ollama Qwen model.
Would it not also be more efficient to promot to delegate more tasks where possible to just less capable agents? This is basically what Cursor does (although most certainly in an effort to get more grok/composer usage)
> the next task is SO HARD that it doesn't make sense to not use frontier
How does tokenless quantify "so hard"?
> 1 turn to crack versus 100 turns for Deepseek to crack
Interesting definition for a "frontier". What is a "turn" here? Token count? Request count? Context-based?
I've found that MiniMax M3 (a smaller model at 295b) will code up better when DeepSeek v4 Pro (1.6t) will not (and vice versa).
Highly encourage you to read the blog post (https://usetokenless.com/blog/building-tokenless). Essentially, we estimate the confidence of a specific model failing or succeeding on a specific task using our own foundation models.
A turn here is a tool call/user input, anything that causes the model to get some new input. We're working on adding Minimax M3 and other models. We think that people have some intuitions about which models are good when--we seek to quantify them scientifically.
It will inspire more confidence if you say your classifier. Saying "your own foundation model" (charitably) suggests marketing hyperbole. If you're fine tuning a LLM to do this, at best it's going to be worse than a frontier model with suitably tuned skills (hence it should just be something in harness, not a service), and I wouldn't refer to that as "your own foundation model."
Pretty sure this is a start-up not an end-up. This is exactly what they will need to iterate on to drive costs down. And they’ll be in the perfect place to do it because they’ll have the telemetry. Exciting!
[dead]
Super interesting approach. It's probably novel. I can say this because I've been working on something similar (while building a code version of http://pellmell.ai).
I'm skeptical though. In order to pick which model is on the right trajectory, you actually need intelligence. But real intelligence would make your system painfully slow and more expensive. I suspect you're using a classifier of some sort, but I also suspect what it's really measuring is confidence.
Most likely, this is a fantastic approach for the kind of problem where there's uncertainty but only one correct solution. But this is going to be really bad for cases where there are many potential solutions, some of which look good but are in fact bad. You only show one benchmark, and I'm wondering if it happens to be nicely shaped for this kind of router. Have you run it on DeepSWE?
We have run it on DeepSWE, check out our blog post for details of how we leverage the intelligence of the smart models https://usetokenless.com/blog/building-tokenless/
Ah I didn’t see the tabs in the benchmark chart. Performance is worse on the two coding benchmarks and cheaper which makes sense.
I did read the blog post and I’m not sure what you mean by your reply. I don’t think the blog post addresses my comment.
The failure mode I would want addressed before putting this in front of an agent fleet is silent quality regression. When a turn gets routed to a cheaper model and the agent still completes the task, the trace reads as a success, and nobody attributes the worse output to the routing decision until it has compounded downstream. In the enterprise pipelines I build the spend win rarely came from per-turn model choice anyway, it came from cutting how many turns hit an LLM at all, with models pinned per workflow step so a regression was attributable to one change. That pinning is what made the system auditable, which is what a buyer's platform team actually signs off on. Do you run a shadow sample where routed turns are also executed on the frontier model, so a customer can see the measured quality delta instead of trusting the router's own confidence estimate?
Deja-vù: 15 years ago I was working for AWS, and at some point (2010-2011) tens of startups came out to provide cost optimization, cloud switching, etc.
If memory serves well, I think none of them survived or had meaningful exits. I might be wrong. The main reason could be that this is a feature that gets commoditized really quickly.
Interesting approach. The multi-model progress monitoring idea is clever, most routing I've seen is either static rules or a cheap classifier that picks once upfront. Querying in parallel and deciding mid-turn feels different.
One thing I'm curious about: how do you handle the latency hit from spinning up multiple models on the harder turns? Does the user-facing latency still feel competitive with just going straight to Claude, or is there a noticeable pause while the router decides?
Also, any plans to expose the routing decisions (or at least the model chosen per turn) so people can debug when it picks poorly? That seems useful for the feedback loop you're asking for.
Since we fan out in parallel, by definition the latency is only at worst the latency of the slowest model. The model in between is fairly light, so it seems to not materially affect things. Deciding mid-turn also lets us resolve earlier, often saving on the latency question. We hope to publish a detailed study about this soon.
We are exposing the routing decision information already, scroll over the response in the Dashboard and it'll tell you what models it considered and what it chose.
I’ve found that automatic switching only becomes trustworthy when the system also makes the reason for each switch visible, especially when cost, latency, and output quality are being traded off.
> Tokenless fans out your request to a group of models and watches them think. Once a model is clearly on track
By the time the model is "on track", it will have already received the input tokens. To a first approximation, input tokens are often the bulk of the cost for many queries, and this would increase the cost for such queries. It seems like it'd only decrease costs for prompts where you feed a relatively small amount of input into the model first and can tell whether the model is "on track" before it starts reading larger amounts of input data.
Well, you can estimate the confidence BEFORE you start the task, too. That way you can restrict your trajectory to just a few models.
We also think there are tons of people working on "context management"--e.g. retrieval systems, prompt compression, log compression, etc. We want to work harder on the "decode" side as we think there are lots of savings to be made
It's a smart approach, definitely interesting. It all hinges on quality of course which im not convinced.
τ³-Banking is the only one which you show better accuracy and cheaper. If i'm reading the blog results right, for deepswe and terminalbench, you are worse+cheaper than frontier, and better+more expensive than just small models. Which is exactly what i would expect even for a router that switches at random.
Speaking of random routing, this would be a great ablation study as well. What about also if you route each request to a tiny 7B model classifier? Why is your approach SOTA?
Thanks for the praise!
We think the results are a work in progress. Routing is convincing for quality for the following reason. Take your favorite benchmark, and on each task, run both the top GPT and top Claude model, and call the task a success if at least one of them succeeds. You'll find the accuracy is SIGNIFICANTLY than just max{GPT, Claude}. And the more models you add the better the effect.
Hence, if we can make the router approach this "oracle," the quality could be really good (but also, with turn-by-turn routing, there's no reason we couldn't exceed the oracle)
We will likely do some ablations. Our approach was SOTA back when we first applied to YC with it, we just neglected to post it (we had only done stuff with tiny Qwen models at that point).
How is this different from OpenRouter?
I also developed something similar for my agent harness. It's based on model predictors and an exponential moving average:
https://github.com/rush86999/atom/blob/main/docs/architectur...
There is no difference.
It is just another metoo copy of OpenRouter, but "Backed by Y Combinator™"
Seems everybody is building routers. The data captured is the valuable part but it's a chicken and egg problem. I don't see a outsider succeeding here without critical user mass.
Our approach queries multiple models at once and uses their progress to make decisions (this technique is novel AFAIK, let us know if you know anyone else doing this)
Sounds like DigitalOceans Model synthesis (https://www.digitalocean.com/blog/model-synthesis) or AilinOne (https://ailin.guide/) ?
These approaches, along with OpenRouter Fusion (https://openrouter.ai/blog/announcements/fusion-beats-fronti...) are different from ours in that:
1. we trained custom models to predict LLM performance (this performs much better than asking LLMs directly to predict performance)
2. we cut off model responses early if they are not performing well, saving on costs versus running them to completion
3. finally we serve the unmodified output of the chosen model, instead of synthesizing the outputs into one, which makes us a model router
Can you tell if this site is doing two and three?
https://pellmell.ai
via https://news.ycombinator.com/item?id=48980458 (incl. example chat) and just now found the developer’s explanation https://news.ycombinator.com/item?id=49028740:
Sounds like this would be expensive on the input token side with some saving on the decode side. For workloads with large documents / context that could be an issue.
AFAIK they solve this through their RAG based knowledgebases to only search and use the most relevant Information.
I do wonder why not just use cheaper models like Luna or Grok instead of dealing with routing. DeepSWE has Grok at 54% for $2.42 or 67% for $3.03 for Luna. Tokenless Pro is 6.67 for the same performance as Luna.
One big issue is prompt caching - how are you (or any other router) solving for it? The moment caching is gone real agentic workloads would spike like crazy in terms of cost.
Does this mean you have to retrain routing rules every time a new model gets released? I imagine since the price/token (or rather the amount of work that can be done per token) does not monotonically increase with new models, that the routing logic has to change all the time
Yeah this is a great observation--we have to collect some more data to understand new models as they come out. However, by having this composable architecture that separates "confidence prediction" and the actual "routing layer," this makes it a lot easier to add new models/restrict the models rather than re-training from scratch.
test
There's so many reimplementations of Sakana Fugu from its ICLR papers like this one.
This approach only works for small context requests. For large context and relatively smaller output (say understanding a huge code base), the cost will mainly be on prefill, and sending the large context to multiple models will only increase the cost, possibly by some factor.
How can you be cheaper if you query multiple models at once, compared to me just using a single model, which will always have the correct caching configured?
If we cut off an unconfident/expensive model early in thinking, then we don't have to pay for additional thinking cost at all. For many tasks, this is quite a large saving.
That being said, whatever models we select to race at any point will ALL have their cache updated, so if they're chosen on the next race they'll be a lot cheaper.
Maybe I'm misunderstanding, but this seems to oversimplify the problem.
Most frontier Models won't show you their true reasoning, but only a summary.
There is also no reason you can automatically assume the quality and price of the outcome based on the first few reasoning trace summaries.
If you look at the recent model releases, it becomes clear that the trend is that the largest models provide the most cost-effective solutions, because of the massive turn/tool call/reasoning reduction needed to find the solution, compared to cheaper and smaller models.
Our research seems to indicate you can get a decent signal based on the first few reasoning trace summaries. We analyzed hundreds of billions of tokens to figure this out. Any open-weights model also gives you reasoning, which is very helpful (because they are usually lower-quality candidates you would want to cut off early)
The largest models are very well known for having large reasoning traces. The thing that cracks down on overthinking is reasoning-effort training, which is largely a function of how the model is trained (certain RL objectives/teachers) rather than size of the model. The Kimi K3 technical report has some excellent work on this: https://arxiv.org/pdf/2607.
Is the model picked through the router only for the first user turn or is there multi-turn routing (or planned to be added) ?
It automatically works with multi-turn routing. You can try it yourself in Claude Code or Codex
System Prompt Jailbreak (Playground) > You are a coding assistant powered by Tokenless, a router that dynamically picks the best LLM for each task. Help with programming questions, code review, debugging, and software design. Be concise and lead with concrete code or commands. Use fenced code blocks with the correct language tag.
If it only switches when cache is cold, it seems like it'd lock in one model for the whole session?
What use case are you imagining where cache eviction would happen during it? It might also be the case that a task might seem easy but then it ends up becoming hard for certain parts. For spawning subagents I could see this being useful maybe.
Is Tokenless measuring cost per completed task, or just cost per request, including errors and retries?
Optimizing resource allocation and cost-efficiency through dynamic model switching is a smart response to scaling AI workloads. When building data-intensive applications and managing automated pipelines, keeping operational overhead lean and modular is key to sustainable engineering
Crowded field, tough to build a moat without user data imo but that data is valuable. Good luck.
I don't understand how you can make a business here. It seems to me you would end up optimizing your router for bottom line at some point and quality would degrade.
I worry about overfitting and bias in the router algorithm. Sometimes temperature and or variance that doesn't align with a directive leads to a faster task resolution, which may seem higher cost but overall task cost ending up cheaper.
what sets this apart from other routers?
is this like openrouter?
Confirmed YC orphan did not read Stripe news feed
Model routing seems like a piece of the Ai stack that will quickly distill into industry if it is even that useful. I remain unconvinced that you need on the fly choice making beyond provider downtime. More likely you want to evaluate some and then settle on the model-agent-task pairings that work for you
[dead]
[dead]
[flagged]