I thought this was fantastic! Surprised not more people are commenting on this. Is there a reason I am not aware of?
To the author: what happens to my voice after I upload it?
What is your plan moving forward? I am too far left field to understand how to build a business and monetize an open source product like this, even though I found it fun to play around with.
Thanks! There is a model that turns the voice into an embedding that is used to determine the voice. Unlike the STT and TTS, we won't be releasing the weights of this voice cloning model, but we will provide it over an API so that we can do verification and prevent abuse.
edit: Ah yes, and we do not store the voice sample on our server. The voice embedding is cached for 24 hours.
Thank you! The TTS and ASR are our own unreleased models, but we'll open-source them soon :)
The latency is about 500ms once we detect that it's the bot's turn to speak (roughly 200ms for the LLM's time-to-first token and 300ms for the TTS audio to start), plus a variable time for the semantic pause detection (VAD).
If it's clear that you're done talking, like when you ask a question, the model will reply very fast. If you stop mid-sentence as if you have more to say, it will wait for longer to avoid interrupting you.
I thought this was fantastic! Surprised not more people are commenting on this. Is there a reason I am not aware of?
To the author: what happens to my voice after I upload it? What is your plan moving forward? I am too far left field to understand how to build a business and monetize an open source product like this, even though I found it fun to play around with.
Thanks! There is a model that turns the voice into an embedding that is used to determine the voice. Unlike the STT and TTS, we won't be releasing the weights of this voice cloning model, but we will provide it over an API so that we can do verification and prevent abuse.
edit: Ah yes, and we do not store the voice sample on our server. The voice embedding is cached for 24 hours.
Incredible work. Short, sweet and simple. I hadn't expected to enjoy this as much as I did. I can't wait to see where it goes.
Way more entertaining than I would expect! What TTS and ASR models do you use? What sort of latency do you get?
Thank you! The TTS and ASR are our own unreleased models, but we'll open-source them soon :)
The latency is about 500ms once we detect that it's the bot's turn to speak (roughly 200ms for the LLM's time-to-first token and 300ms for the TTS audio to start), plus a variable time for the semantic pause detection (VAD).
If it's clear that you're done talking, like when you ask a question, the model will reply very fast. If you stop mid-sentence as if you have more to say, it will wait for longer to avoid interrupting you.