We stream DeepSeek-V4.1-Flash (518 GB, 4-bit) from NVMe on a 128 GB M5 Max, because it doesn't fit in RAM.
prompt processing steady decode
upstream, internal 16.23 10.59
ours, internal only 28.04 1.73x 14.38 1.36x
ours, +1 external 36.88 2.27x 16.05 1.52x
ours, +2 external 43.62 2.69x 17.38 1.64x
The first fork row is the one that matters: same single internal SSD, no replicas, no enclosures.
Just so you know, you are getting automatically flagged in HN (your comments are automatically removed). I think it's because you are using LLM to write them. You should not copy paste LLM output directly to comments in this website.
(I'm not talking about using LLM to generate the README.md of your project, that's also distasteful but not nearly as bad)
In either case your repository is interesting, and might be useful for people that own a M5 Max
We stream DeepSeek-V4.1-Flash (518 GB, 4-bit) from NVMe on a 128 GB M5 Max, because it doesn't fit in RAM.
Just so you know, you are getting automatically flagged in HN (your comments are automatically removed). I think it's because you are using LLM to write them. You should not copy paste LLM output directly to comments in this website.
(I'm not talking about using LLM to generate the README.md of your project, that's also distasteful but not nearly as bad)
In either case your repository is interesting, and might be useful for people that own a M5 Max