I did this manually back in Feb 2024[1], but only got "connector A" mapped out and a partially designed SXM2-to-PCIe interposer designed before getting distracted.
During the process, I discovered Facebook's Big Basin JBOG schematics/specs (from Open Compute), which contained a design derived from NVIDIA's DGX-1 reference implementation. Looks like this clears up some gaps in the current reference. It also explains some of the design choices, like the form factor and similar pinout to OpenPOWER.
Nice, I did a similar thing with OCP documents and an SXM2 server schematic set I found, but ultimately lost interest in the project after I rented a server with 8x V100s and realized it was kind of crap for LLM inference.
I have 4 of these now of the V100S variety. They are aging but solid. If you are not going above 4 cards you probably don’t need SXM2, but the packaging can be easier to deal with. I have PCI variants and network them with 25Gbps and do RDMA.
I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.
It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)
Good to see this re-posted, I posted it two days ago but it sank without a trace so I'm really happy you posted it again. These V100's are a pretty neat GPU and one of very few that is still - somewhat - affordable.
Might be worth depending on the price but GeForce Nvidia GPUs tend to be a better deal than the server class since their EULA doesn’t allow them to be put in datacenters.
Ha, amazing!
I did this manually back in Feb 2024[1], but only got "connector A" mapped out and a partially designed SXM2-to-PCIe interposer designed before getting distracted.
During the process, I discovered Facebook's Big Basin JBOG schematics/specs (from Open Compute), which contained a design derived from NVIDIA's DGX-1 reference implementation. Looks like this clears up some gaps in the current reference. It also explains some of the design choices, like the form factor and similar pinout to OpenPOWER.
https://docs.google.com/spreadsheets/d/1DfH0T9KWWoc2CpKZsRIT...
Nice, I did a similar thing with OCP documents and an SXM2 server schematic set I found, but ultimately lost interest in the project after I rented a server with 8x V100s and realized it was kind of crap for LLM inference.
I have 4 of these now of the V100S variety. They are aging but solid. If you are not going above 4 cards you probably don’t need SXM2, but the packaging can be easier to deal with. I have PCI variants and network them with 25Gbps and do RDMA.
I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.
It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)
Is your engine public?
What rumtime are you using?
Good to see this re-posted, I posted it two days ago but it sank without a trace so I'm really happy you posted it again. These V100's are a pretty neat GPU and one of very few that is still - somewhat - affordable.
Is v100 still good enough for local LLM. I found some cheap 16GB variant, but my clanker told me it's too old for modern llm.
Might be worth depending on the price but GeForce Nvidia GPUs tend to be a better deal than the server class since their EULA doesn’t allow them to be put in datacenters.