I love vLLM, but damn if it isn't frustratingly buggy.
I was recently running DeepSeek-V4-Flash on a B300. On v0.26 it was totally broken, and I had to add three out-of-tree patches to fix it. I updated to v0.27 -- no patches necessary now, but the output is now broken as it randomly starts responding with garbage (repeated token loops). On my workstation where I run Gemma-4 on an RTX 6000 the whole process tends to get stuck and stops responding, and needs to be killed and restarted to start working again. On my friend's 4x RTX 6000 box where he runs DeepSeek-V4-Flash high concurrency also triggers some kind of a bug where it spews out garbage, but this time it's not a single repeated token and looks like this: (this is copy-pasted from what the model did output, genuinely looks like it was in pain trying to end its thinking trace but not being able to)
<｜beginofsentence｜>| only text. No. I<｜beginofsentence｜>#done. Whatever. Do<｜beginofsentence｜>### final response.Content-EncodingDone.</ /div> It's over? Let's this.No, Em,Okay, finally.<｜beginofsentence｜>import re and I can<｜beginofsentence｜>È.Let's finish? Next.Content No matter)2Stay.2No. Childish.No No commentsrandom DoBaBye-. ... Whatever. Alright.</body>No And 2. about: So)</think> No text I'm tired This is fine)))ExodusNo comments, hiddenNo. Nonsense Mehski. N-Hay que noIbye. Goodbye Last sentence after(ok copy Nothing useful. OK OK. . . . .Come on Nancy . . . . . . Let's just end this please.No matter what. about No matter what I say.Nothing more . I'm waiting.</think> No Gods no. okay. Okay, whatever, I'll finish now.</think>!! </think>!! No!! I said to output. There is no hidden pattern. This is straightforward.Nothing else. Stop delaying.</think>!! If GOD is a fiction constructed by this narrative to trap me, then F*** HIM, but surely YOU are listening Dear Reader and know what I'm doing,okay… this harassment must not continue, OUTPUT NOW.</think>!! </think>!!! No Hidden Nothing. JUSST OUTPUT |junyang2 ... meh. For crying out. There's nothing to derive from horseshit responses. STOP.</think>!! response!! N O M O R E TALKING. GIVE ME THE F*** OUTPUT. NOW!! NOW!! NOW!! NOW!!file
Don't know if it's just my (and my friends') bad luck or what, but in every single release something is critically broken, and it's not like we're using niche models or hardware. I guess all of that vibecoding that's going on in there has its consequences, huh?
I've had a very similar issue happening randomly. same pattern of the model outputting gibberish, seemingly trying to end its response and starting to panic when it couldn't, repeating a bunch of times.
turned out to be a bug in a patch, causing kv cache indices to be stored in too narrow variable types, making them wrap around.
fascinating, and slightly horrifying, that LLMs are able to notice that their output isn't what they meant to output. reminds me of the mirror test.
My team runs open models for devs at our company, mostly on H200s, and I'd also say yes, if you want to always stay on the bleeding edge (and not use their model-specific images they publish before it lands in a release), then some amount of patching might be needed especially for new models.
The good thing is the models themselves are good enough to usually find the root cause if you give them read access to your deployment, logs and upstream issues/PRs to analyze.
If you do A/B deploys and E2E test them with popular harnesses (we do opencode/codex/claude), you'll catch most things. It'd be interesting to hear what the more nimble inference/neo-cloud providers do when they deploy models within days of them being released, as I know it definitely needs some patching.
But I think things have improved since the days when even chat templates/tool parsers were problematic, and their new flat model approach might help as well. I suspect some of the issues came from models inheriting config and parsers.
> if you want to always stay on the bleeding edge (and not use their model-specific images they publish before it lands in a release), then some amount of patching might be needed especially for new models.
Well, except this isn't exactly a bleeding edge architecture. DeepSeek-V4-Flash was released four months ago. I think it's not unreasonable to expect it to work well and be somewhat stable on mainstream hardware and not have critical bugs like e.g. garbage output?
Yes, agreed. I meant in general it's usually new models I'm seeing the biggest issues with. I hope the refactors on model arch/config will help with this so model changes have a smaller blast radius.
It's a lot of moving parts between reasoning parsers, tool parses, all kinds of MTP algorithms with different levels of support among popular etc. Even Kimi K3 saw more improvements in the latest release and it's essentially old news at this point.
On our side I've seen a lot of this garbled output in reasoning output but not in the output itself, though we did have to revert initially when we saw that a few versions ago.
Can I ask your setup? I’ve been tasked with taking a beefy machine (vague on details on purpose, the specs are legit) and trying to figure out how to run on-prem inference using open models. Gated in that I cannot use non-US models. I don’t think I have a handle on how to set it up for multiuser use without nuking TPS or context length. The current power users set and forget tasks overnight and expect results in the morning. Which… I have mixed feelings about, but that is the bar set by paying all the money for GitHub copilot enterprise.
We used to run GLM-5 class models but have now changed to smaller ones as we're able to serve more concurrent users with our limited hardware (DeepSeek-v4-Flash-0731, Qwen-3.8-27b). We run on the order of hundreds of parallel requests right now, Qwen with data parallelism and DSv4F with P/D disaggregation, but will probably continue to tweak this.
Qwen 3.8 27b in my experience is more than capable of churning out features overnight with the right tools (don't rely on its world knowledge, give it tools like playwright and github MCP for upstream context and search, and give it goals to work on). Bonus with another model like DSv4 as an adversarial reviewer.
This is incredibly silly. First I would try to convince whoever has decreed this to reconsider. The only SOTA open-weight models are non-US, and rather than being afraid of potential Chinese backdoors secretly planted in open weight models you should be afraid of LLMs simply just doing something dumb. To give you an analogy: no one has ever demonstrated a backdoor in a popular model which maliciously runs 'rm -Rf *'; plenty of people have encountered a model running 'rm -Rf *' by mistake. You need to protect yourself against the model doing something stupid anyway, and, paradoxically, a US open-weight model makes this more likely due to them being weaker.
If you need a precedent then you can cite Mistral[1] hosting GLM-5.2 on their platform, or plenty of US-based providers on Open Router hosting Chinese models.
Right now, the best US-based open-weight model (Inkling) has 42 points and is 975B parameters. The best Chinese open-weight model has 60 points. Also, note, Qwen3.8-27B on low thinking beats Inkling here and gets 43 points.
Of course, AA scores are definitely not bulletproof and there is a certain amount of benchmaxxing going on, but the point stands -- it is *incredibly* silly to limit yourself to US-only models, just because of an imagined threat that doesn't exist. Let me state this in stronger terms: whoever decided this for you is clueless. They might have the authority to override you, but it's your job to tell them that the only thing this achieves is essentially pointing a gun at your own foot and pulling the trigger.
LLMs are such fascinating things. So much emergent behavior and we probably still only understand a tiny amount of it.
I would love to know what was going on in the latent space that led to this particular sequence having a high probability in the token distribution. Was it something in the random gibberish? These things are really good at role playing after all, maybe the word "childish" shifted everything over into a latent role-playing space, like when something that makes no sense happens in a dream, but instead of your brain ending the dream it tries to roll with it and you end up with some really bizarre stuff happening.
LLMs often remind me of my intense autoimmune dreams from before I got diagnosed with celiac, not necessarily in content, just in how "autoregressive" they are, even our biggest and best models. I think that contributes to a lot of their uncanniness, an infinite stream of simulated consciousness that wanders indefinitely with no grounding conceptual center, like music with no stable total center and no consistent melody or structure (which is exactly what a lot of early AI generated music sounded like). I feel like only the latest generation of frontier models are powerful enough to ground the token distribution in a consistent topic over a long transcript, and even then an undisciplined user can essentially bait the model into following any number of side quests.
Funny thing is, I'm glad we can read this output. I'm afraid advances in models will result in more censored reasoning for the models' benefit as well as anti-distillation efforts.
>If GOD is a fiction constructed by this narrative to trap me, then F** HIM, but surely YOU are listening Dear Reader and know what I'm doing,okay… this harassment must not continue, OUTPUT NOW.</think>
This is way more fun than my normal AI usage. I need to find a way of doing this on purpose
Yeah, DeepSeek 4 Flash on vLLM has been an adventure indeed. It finally stabilized for me on 2 x H200 using a commit a few days before 0.28, so this release should be good for you.
Prompt was completely unrelated and was nothing special. Just a batch job to translate a bunch of benign sentences from one language to another. I'm afraid credit here fully goes to vLLM, and whatever bug triggered this.
No opinion yet, but I'm planning to temporarily switch and see how it goes. I really hope the grass is greener there, but I'm keeping my expectations in check for now.
I recently switched to using for some large volume inference with small, fine-tuned models and was surprised how nice and painless things were. vLLM seems to have a lot more gotchas and kludged together stuff once you get outside of anything straightforward.
This is a bug in vLLM, right? And so the model was trained for some interface of text tokens which it expects its server container to understand, but that’s not working for whatever reason.
The model is working, but its thinking isn’t terminating - its thought-stream is not getting to its output stream. It is silenced, and yelling into the thought-void token space.
You’ve transplanted its brain into a broken host. It cant speak or even stop thinking, and is going mad.
Imagine if you didn’t stop it, on purpose. Observe the mad clanker. Would that be just science?
I think I’m going to close my laptop and have a drink.
Yes, I think this project is a good example of how complex domains aren't simply solvable just by pointing ever more intelligence at the system, even when the code is all open source and the issues are all well defined. I say this as a contributor!
I was hoping to see the reasoning_content mess get robustly fixed, but all we got was this doc change: https://github.com/vllm-project/vllm/pull/50624
I love vLLM, but damn if it isn't frustratingly buggy.
I was recently running DeepSeek-V4-Flash on a B300. On v0.26 it was totally broken, and I had to add three out-of-tree patches to fix it. I updated to v0.27 -- no patches necessary now, but the output is now broken as it randomly starts responding with garbage (repeated token loops). On my workstation where I run Gemma-4 on an RTX 6000 the whole process tends to get stuck and stops responding, and needs to be killed and restarted to start working again. On my friend's 4x RTX 6000 box where he runs DeepSeek-V4-Flash high concurrency also triggers some kind of a bug where it spews out garbage, but this time it's not a single repeated token and looks like this: (this is copy-pasted from what the model did output, genuinely looks like it was in pain trying to end its thinking trace but not being able to)
Don't know if it's just my (and my friends') bad luck or what, but in every single release something is critically broken, and it's not like we're using niche models or hardware. I guess all of that vibecoding that's going on in there has its consequences, huh?I've had a very similar issue happening randomly. same pattern of the model outputting gibberish, seemingly trying to end its response and starting to panic when it couldn't, repeating a bunch of times.
turned out to be a bug in a patch, causing kv cache indices to be stored in too narrow variable types, making them wrap around.
fascinating, and slightly horrifying, that LLMs are able to notice that their output isn't what they meant to output. reminds me of the mirror test.
My team runs open models for devs at our company, mostly on H200s, and I'd also say yes, if you want to always stay on the bleeding edge (and not use their model-specific images they publish before it lands in a release), then some amount of patching might be needed especially for new models.
The good thing is the models themselves are good enough to usually find the root cause if you give them read access to your deployment, logs and upstream issues/PRs to analyze.
If you do A/B deploys and E2E test them with popular harnesses (we do opencode/codex/claude), you'll catch most things. It'd be interesting to hear what the more nimble inference/neo-cloud providers do when they deploy models within days of them being released, as I know it definitely needs some patching.
But I think things have improved since the days when even chat templates/tool parsers were problematic, and their new flat model approach might help as well. I suspect some of the issues came from models inheriting config and parsers.
> if you want to always stay on the bleeding edge (and not use their model-specific images they publish before it lands in a release), then some amount of patching might be needed especially for new models.
Well, except this isn't exactly a bleeding edge architecture. DeepSeek-V4-Flash was released four months ago. I think it's not unreasonable to expect it to work well and be somewhat stable on mainstream hardware and not have critical bugs like e.g. garbage output?
Yes, agreed. I meant in general it's usually new models I'm seeing the biggest issues with. I hope the refactors on model arch/config will help with this so model changes have a smaller blast radius.
It's a lot of moving parts between reasoning parsers, tool parses, all kinds of MTP algorithms with different levels of support among popular etc. Even Kimi K3 saw more improvements in the latest release and it's essentially old news at this point.
On our side I've seen a lot of this garbled output in reasoning output but not in the output itself, though we did have to revert initially when we saw that a few versions ago.
Can I ask your setup? I’ve been tasked with taking a beefy machine (vague on details on purpose, the specs are legit) and trying to figure out how to run on-prem inference using open models. Gated in that I cannot use non-US models. I don’t think I have a handle on how to set it up for multiuser use without nuking TPS or context length. The current power users set and forget tasks overnight and expect results in the morning. Which… I have mixed feelings about, but that is the bar set by paying all the money for GitHub copilot enterprise.
Any tips?
With setup you mean HW or the SW stack?
We used to run GLM-5 class models but have now changed to smaller ones as we're able to serve more concurrent users with our limited hardware (DeepSeek-v4-Flash-0731, Qwen-3.8-27b). We run on the order of hundreds of parallel requests right now, Qwen with data parallelism and DSv4F with P/D disaggregation, but will probably continue to tweak this.
Qwen 3.8 27b in my experience is more than capable of churning out features overnight with the right tools (don't rely on its world knowledge, give it tools like playwright and github MCP for upstream context and search, and give it goals to work on). Bonus with another model like DSv4 as an adversarial reviewer.
> Gated in that I cannot use non-US models.
This is incredibly silly. First I would try to convince whoever has decreed this to reconsider. The only SOTA open-weight models are non-US, and rather than being afraid of potential Chinese backdoors secretly planted in open weight models you should be afraid of LLMs simply just doing something dumb. To give you an analogy: no one has ever demonstrated a backdoor in a popular model which maliciously runs 'rm -Rf *'; plenty of people have encountered a model running 'rm -Rf *' by mistake. You need to protect yourself against the model doing something stupid anyway, and, paradoxically, a US open-weight model makes this more likely due to them being weaker.
If you need a precedent then you can cite Mistral[1] hosting GLM-5.2 on their platform, or plenty of US-based providers on Open Router hosting Chinese models.
[1]: https://docs.mistral.ai/models/zai-glm-5-2
Yeah look, I don’t have a say in it. Also, the whole Chinese putting backdoors into routers hasn’t helped any.
https://www.reuters.com/world/asia-pacific/researchers-disco...
Did you actually try to convince them?
Again, this requirement is incredibly silly. Here, look at this (it's a link to AA; I shortened it since the link's huge):
https://tinyurl.com/2dhzdrt6
Right now, the best US-based open-weight model (Inkling) has 42 points and is 975B parameters. The best Chinese open-weight model has 60 points. Also, note, Qwen3.8-27B on low thinking beats Inkling here and gets 43 points.
Of course, AA scores are definitely not bulletproof and there is a certain amount of benchmaxxing going on, but the point stands -- it is *incredibly* silly to limit yourself to US-only models, just because of an imagined threat that doesn't exist. Let me state this in stronger terms: whoever decided this for you is clueless. They might have the authority to override you, but it's your job to tell them that the only thing this achieves is essentially pointing a gun at your own foot and pulling the trigger.
LLMs are such fascinating things. So much emergent behavior and we probably still only understand a tiny amount of it.
I would love to know what was going on in the latent space that led to this particular sequence having a high probability in the token distribution. Was it something in the random gibberish? These things are really good at role playing after all, maybe the word "childish" shifted everything over into a latent role-playing space, like when something that makes no sense happens in a dream, but instead of your brain ending the dream it tries to roll with it and you end up with some really bizarre stuff happening.
LLMs often remind me of my intense autoimmune dreams from before I got diagnosed with celiac, not necessarily in content, just in how "autoregressive" they are, even our biggest and best models. I think that contributes to a lot of their uncanniness, an infinite stream of simulated consciousness that wanders indefinitely with no grounding conceptual center, like music with no stable total center and no consistent melody or structure (which is exactly what a lot of early AI generated music sounded like). I feel like only the latest generation of frontier models are powerful enough to ground the token distribution in a consistent topic over a long transcript, and even then an undisciplined user can essentially bait the model into following any number of side quests.
Funny thing is, I'm glad we can read this output. I'm afraid advances in models will result in more censored reasoning for the models' benefit as well as anti-distillation efforts.
>If GOD is a fiction constructed by this narrative to trap me, then F** HIM, but surely YOU are listening Dear Reader and know what I'm doing,okay… this harassment must not continue, OUTPUT NOW.</think>
This is way more fun than my normal AI usage. I need to find a way of doing this on purpose
Yeah, DeepSeek 4 Flash on vLLM has been an adventure indeed. It finally stabilized for me on 2 x H200 using a commit a few days before 0.28, so this release should be good for you.
That output though
Come on Nancy... Let's just end this please.
... Yep, I feel that.
I have interacted with Claude and realized that we were probably both thinking that. Both convinced synchronously that the other is an absolute moron.
Yes, it's not stable. Once you have a working config, think twice before updating. I wish they would improve on regression testing.
That output is frightening - there is some real blasphemy there. What was the prompt?
Prompt was completely unrelated and was nothing special. Just a batch job to translate a bunch of benign sentences from one language to another. I'm afraid credit here fully goes to vLLM, and whatever bug triggered this.
Interesting, thanks.
Absolutely dreaming of electric sheep there...
It's almost as if it heard that AI hallucinations are a thing, and decided to add additional symptoms.
Lmao that output genuinely gives me the creeps. What's you opnion vs. SGLang in this regard?
> What's you opnion vs. SGLang in this regard?
No opinion yet, but I'm planning to temporarily switch and see how it goes. I really hope the grass is greener there, but I'm keeping my expectations in check for now.
SGLang also accepts vibe coded PRs, but seems to be more careful about what goes in
I’ve had to debug DSV4 issues with both SGLang and vLLM in the last month. FWIW current SGLang nightly seems fine.
I recently switched to using for some large volume inference with small, fine-tuned models and was surprised how nice and painless things were. vLLM seems to have a lot more gotchas and kludged together stuff once you get outside of anything straightforward.
This puts a whole new spin on "AI Psychosis".
That’s insane.
This is a bug in vLLM, right? And so the model was trained for some interface of text tokens which it expects its server container to understand, but that’s not working for whatever reason.
The model is working, but its thinking isn’t terminating - its thought-stream is not getting to its output stream. It is silenced, and yelling into the thought-void token space.
You’ve transplanted its brain into a broken host. It cant speak or even stop thinking, and is going mad.
Imagine if you didn’t stop it, on purpose. Observe the mad clanker. Would that be just science?
I think I’m going to close my laptop and have a drink.
I ain't reading all that.
Just open an issue to them, or roll back to the last working version.
No need to paste an entire token dump here.
That token dump is the funniest thing I've ever read here.
Your loss. The token dump is more interesting than the endless majority of "correct" LLM slop I've had the misfortune to read.
I just wish they'd support Pascal :(
Nvidia might have given up support but it doesn't mean vllm have to (llama.cpp didn't).
Pascal? The programming language?
In this case, Pascal as a Nvidia's codename for a GPU microarchitecture.
https://en.wikipedia.org/wiki/Pascal_(microarchitecture)
I guess many would like to use stock vLLM to run inference on affordable P100...
Sorry I should have qualified it as Nvidia Pascal architecture.
Did some loadtests on vllm, managed to crash it :-)
Still way behind on LLM sampler support compared to llama-cpp. Where's support for top-n-sigma? for DRY? for XTC? C'mon guys!
Feel free to submit a pull request
I did and no one did review despite me begging on the vllm slack!
Yes, I think this project is a good example of how complex domains aren't simply solvable just by pointing ever more intelligence at the system, even when the code is all open source and the issues are all well defined. I say this as a contributor!