I recommend reading some foundational materials on federated learning and distributed inference. The fundamental issues are (1) compute (2) latency and (3) security.
On the compute side, you can't load most individual experts into even high-end consumer-grade cards (e.g. RTX 5090) because the weights are simply too big.
On the latency side, you need to ship the activations and wait for the somewhat underpowered cards to actually do the matrix multiplications, then send the results back. This means each token takes hundreds of milliseconds or even more, which is borderline unusable.
Finally if you're running a P2P network, the only way to verify that peers aren't lying to you is to run duplicate calculations - I don't see how SHA-256 or signed model state help here unless I'm missing something (What are you calculating the hash of? What exactly is signing attesting to?).
This is one of the few places where borrowing ideas from cryptocurrencies actually makes sense - you could imagine a system where peers place bonds and forfeit them if they lie about calculations. You can look at projects like Bittensor and others for how these kind of things are currently implemented.
Thanks for the comment. Extremely clear and detailed. To write this project, I researched a lot about methods and how peers should work compared to a server, as well as security concepts, which certainly have more to add. The goal for this type of project was to move from a single-machine Colibri to multiple-machine Lumabri on a LAN to a large number of machines working together in a Napster-style P2P Lumabri. For latency and other issues, we are studying every type of method that can improve it, and we are also writing and testing other things on our test server. Thank you very much, we will continue to improve.
Actually, no, in this case it would be possible that if other peers in the network give up processing or space you would have a speed that you wouldn't have as a single computer.
There needs to be some sort of safeguard to prevent people from abusing this and running queries to do something an owner objects to. I'd feel horrible if I discovered that someone used my AI to figure out how to kill someone. Another problem would be privacy. My highly personal queries would be available to whomever ran that query for me thru logging.
This is not to say it's a bad idea, but it needs some thought toward protecting users or at least acknowledgment and warnings that their information might be available to unintended eyes.
absolutely controls and other things will be part of everything for this type of project.
Thanks for the comment and the goal is definitely to improve it more and more.
Adding information for this type of project might make the readme seem overloaded with information, but the goal was to show each test and explain it as best as possible. I'll work on it, thank you very much.
I recommend reading some foundational materials on federated learning and distributed inference. The fundamental issues are (1) compute (2) latency and (3) security.
On the compute side, you can't load most individual experts into even high-end consumer-grade cards (e.g. RTX 5090) because the weights are simply too big.
On the latency side, you need to ship the activations and wait for the somewhat underpowered cards to actually do the matrix multiplications, then send the results back. This means each token takes hundreds of milliseconds or even more, which is borderline unusable.
Finally if you're running a P2P network, the only way to verify that peers aren't lying to you is to run duplicate calculations - I don't see how SHA-256 or signed model state help here unless I'm missing something (What are you calculating the hash of? What exactly is signing attesting to?).
This is one of the few places where borrowing ideas from cryptocurrencies actually makes sense - you could imagine a system where peers place bonds and forfeit them if they lie about calculations. You can look at projects like Bittensor and others for how these kind of things are currently implemented.
Thanks for the comment. Extremely clear and detailed. To write this project, I researched a lot about methods and how peers should work compared to a server, as well as security concepts, which certainly have more to add. The goal for this type of project was to move from a single-machine Colibri to multiple-machine Lumabri on a LAN to a large number of machines working together in a Napster-style P2P Lumabri. For latency and other issues, we are studying every type of method that can improve it, and we are also writing and testing other things on our test server. Thank you very much, we will continue to improve.
LLMs are already slow on one machine. Now you are going to add network calls to it?
Actually, no, in this case it would be possible that if other peers in the network give up processing or space you would have a speed that you wouldn't have as a single computer.
There needs to be some sort of safeguard to prevent people from abusing this and running queries to do something an owner objects to. I'd feel horrible if I discovered that someone used my AI to figure out how to kill someone. Another problem would be privacy. My highly personal queries would be available to whomever ran that query for me thru logging.
This is not to say it's a bad idea, but it needs some thought toward protecting users or at least acknowledgment and warnings that their information might be available to unintended eyes.
absolutely controls and other things will be part of everything for this type of project. Thanks for the comment and the goal is definitely to improve it more and more.
Here is some feedback: don't put slop in your readme. It's unreadable.
Adding information for this type of project might make the readme seem overloaded with information, but the goal was to show each test and explain it as best as possible. I'll work on it, thank you very much.