Interesting, the pacing seemed very slow when conversing in english, but when I spoke to it in spanish, it sounded much faster. It's really impressive that these models are going to be able to do real time translation and much more.
The Chinese are going to end up owning the AI market if the American labs don't start competing on open weights. Americans may end up in a situation where they have some $1000-2000 device at home with an open Chinese model running on it, if they care about privacy or owning their data. What a turn of events!
sitting here in the US, reading that China is strongly urging the adoption of Linux and pushing for open CPU architectures like RISC-V and also self-hosted open models
This is exactly what I do. I have two 3090s at home, with Qwen3 on it. This is tied into my Home Assistant install, and I use esp32 devices as voice satellites. It works shockingly well.
I run Home Assistant on an RPi4 and have an ESP32-based Core2 with mic (https://shop.m5stack.com/products/m5stack-core2-esp32-iot-de...), along with a 16GB 4070 Ti Super in an always-on Windows system I only use for occasional gaming and serving media. I'd love to set up something like you have. Can you recommended a starting place, or ideally, a step-by-step tutorial?
I've never set up any AI system. Would you say setting up such a self-hosted AI is at a point now where an AI novice can get an AI system installed and integrated with an existing Home Assistant install in a couple hours?
I mean - the AI itself will help you get all that setup.
Claude code is your friend.
I run proxmox on an old Dell R710 in my closet that hosts my homeassistant (amongst others) VM and then I've setup my "gaming" PC (which hasn't done any gaming in quite some time) to dual boot (Windows or Deb/Proxmox) and just keep it booted into Deb as another proxmox node. That PC also has a 4070 Super that I have setup to passthru to a VM and on that VM I've got various services utilizing the GPU. This includes some that are utilized by my hetzner bare metal servers for things like image/text embeddings as well as local LLM use (though, rather minimal due to VRAM constraints) and some image/video object detection stuff with my security cameras (slowly working on a remote water gun turret to keep the racoons from trying to eat the kittens that stray cats keep having in my driveway/workshop).
Install claude code (or, opencode, it's also good) - use Opus (get the max plan) and give it a directory that it can use as it's working directory (don't open it in ~/Documents and just start doing things) and prompt it with something as simple as this:
"I have an existing home assistant setup at home and I'd like to determine what sort of self-hosted AI I could setup and integrate with that home assistant install - can you help me get started? Please also maintain some notes in .md files in this working directory with those note files named and organized as you see appropriate so that we can share relevant context and information with future sessions. (example: Hardware information, local urls, network layout, etc) If you're unsure of something, ask me questions. Do not perform any destructive actions without first confirming with me."
Plan mode. _ALWAYS_ use plan mode to get the task setup, if there's something about the plan you don't like, say no and give it notes - it will return with a new plan. Eventually agree to the plan when it's right - then work through that plan not in plan mode, but if it gets off the plan, get back in plan mode to get the/a plan set and then again let it go and just steer it in regular mode.
I assume it's very similar to what Home Assistant's backing commercial entity Nabu Casa sells with the "Home Assistant Voice PE" device, which is also esp32-based. The code is open and uses the esphome framework so it's fairly easy to recreate on custom HW you have laying around.
That's great to hear. I was mostly impressed with Qwen3 coder on my 4090, but am hobbled by the small memory footprint of the single card. What motherboard are you using with your 3090s? Like the others, I too am curious about those esp32s and what software you run on them.
Keep up the good hacking - it's been fun to play with this stuff!
He is referring the M5 Atom's I believe. I strongly recommend the ESP32 S3 box now, you can fire up Bobbas special firmware for it, search on Github, and its a blast with Home Assistant.
When has the average American ever been willing to spend a $1,000-2,000 premium for privacy-respecting tech? They already save $20-200 to buy IoT cameras which provide all audio and video from inside their home directly to the government without a warrant (Ring vs Reolink/etc).
To be fair, it isn't $1000-2000 extra, it's the new laptop/pc you just bought that is powerful enough (now, or in the near future) to run these open weight models.
Wiredpancake got flagged to death but they’re right. MacWhisper provides a great example of good value for dead-simple user-friendly on-device processing.
> Americans may end up in a situation where they have some $1000-2000 device at home with an open Chinese model running on it, if they care about privacy or owning their data.
I think HN vastly overestimates the market for something like this. Yes, there are some people who would spend $2,000 to avoid having prompts go to any cloud service.
However, most people don’t care. Paying $20 per month for a ChatGPT subscription is a bargain and they automatically get access to new versions as they come.
I think the at-home self hosting hobby is interesting, but it’s never going to be a mainstream thing.
There is going to be a big market for private AI appliances, in my estimation at least.
Case in point: I give Gmail OAuth access to nobody. I nearly got burned once and I really don’t want my entire domain nuked. But I want to be able to have an LLM do things only LLMs can do with my email.
“Find all emails with ‘autopay’ in the subject from my utility company for the past 12 months, then compare it to the prior year’s data.” GPT-OSS-20b tried its best but got the math obviously wrong. Qwen happily made the tool calls and spat out an accurate report, and even offered to make a CSV for me.
Surely if you can’t trust npm packages or MS to not hand out god tokens to any who asks nicely, you shouldn’t trust a random MCP server with your credentials or your model. So I had Kilocode build my own. For that use case, local models just don’t quite cut it. I loaded $10 into OpenRouter, told it what I wanted, and selected GPT5 because it’s half off this week. 45 minutes, $0.78, and a few manual interventions later I had a working Gmail MCP that is my very own. It gave me some great instructions on how to configure an OAuth app in GCP, and I was able to get it running queries within minutes from my local models.
There is a consumer play for a ~$2499-$5000 box that can run your personal staff of agents on the horizon. We need about one more generation of models and another generation of low-mid inference hardware to make it commercially feasible to turn a profit. It would need to pay for itself easily in the lives of its adopters. Then the mass market could open up. A more obvious path goes through SMBs who care about control and data sovereignty.
If you’re curious, my power bill is up YoY, but there was a rate hike, definitely not my 4090;).
I agree the market is niche atm, but I can't help but disagree with your outlook long term. Self hosted models don't have the problems ChatGPT subscribers are facing with models seemingly performing worse over time, they don't need to worry about usage quotas, they don't need to worry about getting locked out of their services, etc.
All of these things have a dark side, though; but it's likely unnecessary for me to elaborate on that.
Americans may end up in a situation where they have some $1000-2000 device at home with an open Chinese model running on it
Wouldn't worry about that, I'm pretty sure the government is going to ban running Chinese tech in this space sooner or later. And we won't even be able to download it.
Not saying any of the bans will make any kind of sense, but I'm pretty sure they're gonna say this is a "strategic" space. And everything else will follow from there.
When DeepSeek first hit the news, an American senator proposed adding it to ITAR so they could send people to prison for using it. Didn't pass, thankfully.
For criminal concerns regarding retroactive ITAR additions, yes. However, significant civil financial penalties if congress so wished could still be constitutional as the ex post facto clause has been held to apply exclusively to criminal matters starting in Calder v. Bull [1].
History is littered with unconstitutional, enforced laws, as well. Watched a lot of Ken Burns docs this weekend while sick. “The West” has quite a few examples.
You can try it out on https://chat.qwen.ai/ - sign in with Google or GitHub (signed out users can't use the voice mode) and then click on the voice icon.
It has an entertaining selection of different voices, including:
*Dylan* - A teenager who grew up in Beijing's hutongs
Not quite. The smallest Qwen3 A3B quants are ~12gb and use more like ~14gb depending on your context settings. You'll thrash the SSD pretty hard swapping it on a 16gb machine.
A fun project for somebody who has more time than myself would be to see if they can get it working with the new Mojo stuff from yesterday for Apple. I don't know if the functionality would be fully baked out enough yet to actually do the port successfully, but it would be an interesting try.
Not yet as far as I can tell - might take a while for someone to pull that together given the complexity involved in handling audio and image and text and video at once.
It'd run on a 5090 with 32GB of VRAM at fp8 quantization which is generally a very acceptable size/quality trade-off. (I run GLM-4.5-Air at 3b quantization!) The transformer architecture also lends itself quite well to having different layers of the model running in different places, so you can 'shard' the model across different compute nodes.
Here is the demo video on it. The video w/ sound input -> sound output while doing translation from the video to another language was the most impressive display I've seen yet.
The real point of leverage for here is performance/size. Getting traction in the open weights space kinda forces that the models need to innovate on efficiency. This means the open weight models may get leverage that the closed weight ones don't think about.
If we had some aggregated cluster reasoning mechanisms, When would 8x 30B models running on an h100 server out perform in terms of accuracy 1 240B model on the same server.
Neat. I threw a couple simple audio clips at it and it was able to at least recognize the instrumentation (piano, drums, etc). I haven't seen a lot of multimodal LLM focus around recognizing audio outside of speech, so I'd love to see a deep dive of what the SOTA is.
Any insights into what "native video support" actually means? Is it just good at interpreting consecutive full frame images taken at intervals (thus missing out on fast events) or is there something more elaborate to it?
Speech input + speech output is a big deal. In theory you can talk to it using voice, and it can respond in your language, or translate for someone else, without intermediary technologies. Right now you need wakeword, speech to text, and then text to speech, in addition to your core LLM. A couple can input speech, or output speech, but not both. It looks like they have at least 3 variants in the ~32b range.
Depending on the architecture this is something you could feasibly have in your house in a couple of years or in an expensive "ai toaster"
The opportunities of plugging this into your home automation through tool calls is huge.
Ever since ChatGPT added this feature I've been waiting for anyone else to catch up.
They're are tons of hands free situations like cooking where this would be amazing ("read the next step please, my hands are covered in raw pork", "how much flour for the roux", "crap, I don't have any lemons, what can I substitute")
Seems like a big win for language learning, if nothing else. Also seems possible to run locally, especially once the unsloth guys get their hands on it.
The qwen thinker/speaker architecture is really fascinating and is more in line with how I imagine human multi modality works - IE, a picture of an apple, the text a p p l e and the sound all map to the same concept without going to text first.
The existing vision LLMs all work like this, which is most of the major models these days.
Multi-modal audio models are a lot less common. GPT-4o was meant to be able to do this natively from the start but they ended up shipping separate custom models based on it for their audio features. As far as I can tell GPT-5 doesn't have audio input/output at all - the OpenAI features for that still use GPT-4o-audio.
I don't know if Gemini 2.5 (which is multi-modal for vision and audio) shares the same embedding space for all three, but I expect it probably does.
There are many more weird and complex architectures in models for video understanding.
For example, beyond video->text->llm and video->embedding in llm, you can also have an llm controlling/guiding a separate video extractor.
See this paper for a pretty thorough overview.
Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., Vosoughi, A., Huang, C., Zhang, Z., Liu, P., Feng, M., Zheng, F., Zhang, J., Luo, P., Luo, J., & Xu, C. (2025). Video Understanding with Large Language Models: A Survey (No. arXiv:2312.17432). arXiv. https://doi.org/10.48550/arXiv.2312.17432
Yeah, and that's my understanding. Nothing goes video -> text, or audio -> text, or even text -> text without first going through state space. That's where the core of the transformer architecture is.
I recently needed to scan hundreds of low quality invoices and run them through OCR for invoice numbers and dates. I really took for granted how seamless this is in some applications, and was shocked how much work went into producing decent results.
I was obviously really naive. Either way, it gets me excited any time I see progress with OCR. I should give this a try against my (small) dataset.
- additional modalities
- Faster FPS (inferences per second)
- Reaction time tuning (latency vs quality tradeoff) for visual and audio inputs/outputs
- built-in planning modules in the architecture (think premotor frontal lobe)
- time awareness during inference (towards an always inferring / always learning architecture)
I usually ask these models to tell me a short story, and most times the prose is stiff and the story reads like a mass market straight to KDP kids book. But wow, first shot generated something, light, mildly funny, and chill. Quite a surprise.
Pasted here for your own judgement:
*Title: The Last Lightbulb*
The power had been out for three days. Rain drummed against the windows of the old cabin, and the only light came from a flickering candle on the kitchen table.
Maggie, wrapped in a wool blanket, squinted at the last working flashlight. “We’ve got one bulb left, Jack. One.”
Jack, hunched over a board game he’d dug out of the closet, didn’t look up. “Then don’t turn it on unless you’re reading Shakespeare or delivering a baby.”
She rolled her eyes. “I need it to find the can opener. I’m not eating cold beans with my fingers again.”
Jack finally glanced up, grinning. “You did that yesterday and called it ‘rustic dining.’”
“Desperate times,” she muttered, clicking the flashlight on. The beam cut through the gloom—and immediately began to dim.
“No—!” Jack lunged, but too late. The light sputtered… then died.
Silence. Then Maggie sighed. “Well. There goes civilization.”
Jack leaned back, chuckling. “Guess we’re officially cavemen now.”
“Cavewoman,” she corrected, fumbling in the dark. “And I’m going to bed. Wake me when the grid remembers we exist.”
As she shuffled off, Jack called after her, “Hey—if you find the can opener in the dark, you’re officially magic.”
A pause. Then, from down the hall: “I found socks that match. That’s basically witchcraft.”
Jack smiled into the dark. “Goodnight, witch.”
“Goodnight, caveman.”
Outside, the rain kept falling. Inside, the dark didn’t feel so heavy anymore.
will be interesting how it will compare with pricing with audio modality comparing to gemini 2.0 flash once many providers offer it.
Even though gemini 2.0 flash is quite old I still like it. Very cheap (each second of audio is just 32 tokens), support even more languages, non-reasoning so very fast, big rate limits.
Does this support realtime speech to speech via API? If so, where is this hosted/documented? I wasn’t able to see any info. I’d love to use this in lieu of OAIs (expensive) real time speech to speech offering.
Not really, it's a significant place which is why the protest (and hence massacre) was there, so especially for Chinese people (I expect) merely referencing it doesn't so immediately refer to the massacre, they have plenty of other connotations for it.
e.g. if something similar happened in Trafalgar Square, I expect it would still be primarily a major square in London to me, not oh my god they must be referring to that awful event. (In fact I think it was targeted in the 7/7 bombings for example.)
Or a better example to go with your translation - you can refer to the Bastille without 'boldly' invoking the histoire of its storming in the French Revolution.
No doubt the US media has referred to the Capitol without boldness many times since 6 Jan '21.
Not to mention, Tiananmen Square is one of the major tourist destinations in Beijing (similar to National Mall in Washington DC), for both domestic and foreign visitors.
This is true. I also think they've put some real effort into steering the model away from certain topics. If you ask too closely you'll get a response like:
"As an AI assistant, I must remind you that your statements may involve false and potentially illegal information. Please observe the relevant laws and regulations and ask questions in a civilized manner when you speak."
Interesting, the pacing seemed very slow when conversing in english, but when I spoke to it in spanish, it sounded much faster. It's really impressive that these models are going to be able to do real time translation and much more.
The Chinese are going to end up owning the AI market if the American labs don't start competing on open weights. Americans may end up in a situation where they have some $1000-2000 device at home with an open Chinese model running on it, if they care about privacy or owning their data. What a turn of events!
sitting here in the US, reading that China is strongly urging the adoption of Linux and pushing for open CPU architectures like RISC-V and also self-hosted open models
are we the baddies??
If there is a walled garden, and you aren't in it, you'll probably push for the walls to come down. No moral basis needed.
This is exactly what I do. I have two 3090s at home, with Qwen3 on it. This is tied into my Home Assistant install, and I use esp32 devices as voice satellites. It works shockingly well.
I run Home Assistant on an RPi4 and have an ESP32-based Core2 with mic (https://shop.m5stack.com/products/m5stack-core2-esp32-iot-de...), along with a 16GB 4070 Ti Super in an always-on Windows system I only use for occasional gaming and serving media. I'd love to set up something like you have. Can you recommended a starting place, or ideally, a step-by-step tutorial?
I've never set up any AI system. Would you say setting up such a self-hosted AI is at a point now where an AI novice can get an AI system installed and integrated with an existing Home Assistant install in a couple hours?
I mean - the AI itself will help you get all that setup.
Claude code is your friend.
I run proxmox on an old Dell R710 in my closet that hosts my homeassistant (amongst others) VM and then I've setup my "gaming" PC (which hasn't done any gaming in quite some time) to dual boot (Windows or Deb/Proxmox) and just keep it booted into Deb as another proxmox node. That PC also has a 4070 Super that I have setup to passthru to a VM and on that VM I've got various services utilizing the GPU. This includes some that are utilized by my hetzner bare metal servers for things like image/text embeddings as well as local LLM use (though, rather minimal due to VRAM constraints) and some image/video object detection stuff with my security cameras (slowly working on a remote water gun turret to keep the racoons from trying to eat the kittens that stray cats keep having in my driveway/workshop).
Install claude code (or, opencode, it's also good) - use Opus (get the max plan) and give it a directory that it can use as it's working directory (don't open it in ~/Documents and just start doing things) and prompt it with something as simple as this:
"I have an existing home assistant setup at home and I'd like to determine what sort of self-hosted AI I could setup and integrate with that home assistant install - can you help me get started? Please also maintain some notes in .md files in this working directory with those note files named and organized as you see appropriate so that we can share relevant context and information with future sessions. (example: Hardware information, local urls, network layout, etc) If you're unsure of something, ask me questions. Do not perform any destructive actions without first confirming with me."
Plan mode. _ALWAYS_ use plan mode to get the task setup, if there's something about the plan you don't like, say no and give it notes - it will return with a new plan. Eventually agree to the plan when it's right - then work through that plan not in plan mode, but if it gets off the plan, get back in plan mode to get the/a plan set and then again let it go and just steer it in regular mode.
Seems interesting setup, do you have it documented anywhere, thinking of building one!
Ooo interesting, I'd love to hear more about the esp32's as voice satellites!
I assume it's very similar to what Home Assistant's backing commercial entity Nabu Casa sells with the "Home Assistant Voice PE" device, which is also esp32-based. The code is open and uses the esphome framework so it's fairly easy to recreate on custom HW you have laying around.
Do you also add custom tools to turn on/off the lights?
That's great to hear. I was mostly impressed with Qwen3 coder on my 4090, but am hobbled by the small memory footprint of the single card. What motherboard are you using with your 3090s? Like the others, I too am curious about those esp32s and what software you run on them.
Keep up the good hacking - it's been fun to play with this stuff!
omg, this is something I've had in mind for quite some time, I even bought some i2s devices to test it out. Do you have some pointers on how to do it?
Can you tell me about these voice satellites?
He is referring the M5 Atom's I believe. I strongly recommend the ESP32 S3 box now, you can fire up Bobbas special firmware for it, search on Github, and its a blast with Home Assistant.
When has the average American ever been willing to spend a $1,000-2,000 premium for privacy-respecting tech? They already save $20-200 to buy IoT cameras which provide all audio and video from inside their home directly to the government without a warrant (Ring vs Reolink/etc).
To be fair, it isn't $1000-2000 extra, it's the new laptop/pc you just bought that is powerful enough (now, or in the near future) to run these open weight models.
Ease of use is a major issue.
What percentage of the people that you know are able to install python and dependencies plus the correct open weights models?
I'd wager most of your parents can't do it.
Most "normies" wouldn't even know what a local model even is, let alone how to install a GPU.
Wiredpancake got flagged to death but they’re right. MacWhisper provides a great example of good value for dead-simple user-friendly on-device processing.
That sounds like all software. We can make better software.
> Americans may end up in a situation where they have some $1000-2000 device at home with an open Chinese model running on it, if they care about privacy or owning their data.
I think HN vastly overestimates the market for something like this. Yes, there are some people who would spend $2,000 to avoid having prompts go to any cloud service.
However, most people don’t care. Paying $20 per month for a ChatGPT subscription is a bargain and they automatically get access to new versions as they come.
I think the at-home self hosting hobby is interesting, but it’s never going to be a mainstream thing.
There is going to be a big market for private AI appliances, in my estimation at least.
Case in point: I give Gmail OAuth access to nobody. I nearly got burned once and I really don’t want my entire domain nuked. But I want to be able to have an LLM do things only LLMs can do with my email.
“Find all emails with ‘autopay’ in the subject from my utility company for the past 12 months, then compare it to the prior year’s data.” GPT-OSS-20b tried its best but got the math obviously wrong. Qwen happily made the tool calls and spat out an accurate report, and even offered to make a CSV for me.
Surely if you can’t trust npm packages or MS to not hand out god tokens to any who asks nicely, you shouldn’t trust a random MCP server with your credentials or your model. So I had Kilocode build my own. For that use case, local models just don’t quite cut it. I loaded $10 into OpenRouter, told it what I wanted, and selected GPT5 because it’s half off this week. 45 minutes, $0.78, and a few manual interventions later I had a working Gmail MCP that is my very own. It gave me some great instructions on how to configure an OAuth app in GCP, and I was able to get it running queries within minutes from my local models.
There is a consumer play for a ~$2499-$5000 box that can run your personal staff of agents on the horizon. We need about one more generation of models and another generation of low-mid inference hardware to make it commercially feasible to turn a profit. It would need to pay for itself easily in the lives of its adopters. Then the mass market could open up. A more obvious path goes through SMBs who care about control and data sovereignty.
If you’re curious, my power bill is up YoY, but there was a rate hike, definitely not my 4090;).
I agree the market is niche atm, but I can't help but disagree with your outlook long term. Self hosted models don't have the problems ChatGPT subscribers are facing with models seemingly performing worse over time, they don't need to worry about usage quotas, they don't need to worry about getting locked out of their services, etc.
All of these things have a dark side, though; but it's likely unnecessary for me to elaborate on that.
There is some irony about buying Chinese hardware to run American software on it for the past decade(s), and now the exact reverse.
Is that irony? Regardless, it's hilarious. All of the upvotes to you!
Americans may end up in a situation where they have some $1000-2000 device at home with an open Chinese model running on it
Wouldn't worry about that, I'm pretty sure the government is going to ban running Chinese tech in this space sooner or later. And we won't even be able to download it.
Not saying any of the bans will make any kind of sense, but I'm pretty sure they're gonna say this is a "strategic" space. And everything else will follow from there.
Download Chinese models while you can.
When DeepSeek first hit the news, an American senator proposed adding it to ITAR so they could send people to prison for using it. Didn't pass, thankfully.
If it does in the future, do we just hope it won’t be retroactive? Is this water boiling yet?
ex post facto law is explicitly banned in the US Constitution
For criminal concerns regarding retroactive ITAR additions, yes. However, significant civil financial penalties if congress so wished could still be constitutional as the ex post facto clause has been held to apply exclusively to criminal matters starting in Calder v. Bull [1].
[1] https://www.oyez.org/cases/1789-1850/3us386
Dogs can't play basketball, either, but we've sure been getting dunked on a lot lately.
History is littered with unconstitutional, enforced laws, as well. Watched a lot of Ken Burns docs this weekend while sick. “The West” has quite a few examples.
government hardly has the capacity to ban foreign weights
The danger is that lawmakers, confused about the difference between foreign weights and foreign APIs, accidentally ban both.
Eh this is the internet. There's always a way. They couldn't ban piracy either.
You can try it out on https://chat.qwen.ai/ - sign in with Google or GitHub (signed out users can't use the voice mode) and then click on the voice icon.
It has an entertaining selection of different voices, including:
*Dylan* - A teenager who grew up in Beijing's hutongs
*Peter* - Tianjin crosstalk, professionally supporting others
*Cherry* - A sunny, positive, friendly, and natural young lady
*Ethan* - A sunny, warm, energetic, and vigorous boy
*Eric* - A Sichuan Chengdu man who stands out from the crowd
*Jada* - The fiery older sister from Shanghai
Many of these voices are especially hillarious when you switch the language.
In Russian, Ryan sounds like a westerner who started reading Russian words a month ago.
Dylan sounds somewhat authentic, while everyone else is a different degree of heavy-asian-accented Russian.
The voices are really fun, thanks for the laughs :)
I only see Omni Flash, is that the one?
same, did you figure it out?
I think so, you need to click the big jagged audio icon to start a voice session.
The model weights are 70GB (Hugging Face recently added a file size indicator - see https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct/tree... ) so this one is reasonably accessible to run locally.
I wonder if we'll see a macOS port soon - currently it very much needs an NVIDIA GPU as far as I can tell.
That's at BF16, so it should fit fairly well on 24GB GPUs after quantization to Q4, I'd think. (Much like the other 30B-A3B models in the family.)
I'm pretty happy about that - I was worried it'd be another 200B+.
are there any that would run on 16GB Apple M1?
Not quite. The smallest Qwen3 A3B quants are ~12gb and use more like ~14gb depending on your context settings. You'll thrash the SSD pretty hard swapping it on a 16gb machine.
A fun project for somebody who has more time than myself would be to see if they can get it working with the new Mojo stuff from yesterday for Apple. I don't know if the functionality would be fully baked out enough yet to actually do the port successfully, but it would be an interesting try.
New Mojo stuff from Apple?
Nvm found it https://news.ycombinator.com/item?id=45326388
is there an inference engine for this on macos?
Not yet as far as I can tell - might take a while for someone to pull that together given the complexity involved in handling audio and image and text and video at once.
Would it run on 5090? Or is it possible to link multiple GPUs or has NVIDIA locked it down?
It'd run on a 5090 with 32GB of VRAM at fp8 quantization which is generally a very acceptable size/quality trade-off. (I run GLM-4.5-Air at 3b quantization!) The transformer architecture also lends itself quite well to having different layers of the model running in different places, so you can 'shard' the model across different compute nodes.
Here is the demo video on it. The video w/ sound input -> sound output while doing translation from the video to another language was the most impressive display I've seen yet.
https://www.youtube.com/watch?v=_zdOrPju4_g
The real point of leverage for here is performance/size. Getting traction in the open weights space kinda forces that the models need to innovate on efficiency. This means the open weight models may get leverage that the closed weight ones don't think about.
If we had some aggregated cluster reasoning mechanisms, When would 8x 30B models running on an h100 server out perform in terms of accuracy 1 240B model on the same server.
Neat. I threw a couple simple audio clips at it and it was able to at least recognize the instrumentation (piano, drums, etc). I haven't seen a lot of multimodal LLM focus around recognizing audio outside of speech, so I'd love to see a deep dive of what the SOTA is.
Any insights into what "native video support" actually means? Is it just good at interpreting consecutive full frame images taken at intervals (thus missing out on fast events) or is there something more elaborate to it?
Speech input + speech output is a big deal. In theory you can talk to it using voice, and it can respond in your language, or translate for someone else, without intermediary technologies. Right now you need wakeword, speech to text, and then text to speech, in addition to your core LLM. A couple can input speech, or output speech, but not both. It looks like they have at least 3 variants in the ~32b range.
Depending on the architecture this is something you could feasibly have in your house in a couple of years or in an expensive "ai toaster"
The opportunities of plugging this into your home automation through tool calls is huge.
Ever since ChatGPT added this feature I've been waiting for anyone else to catch up.
They're are tons of hands free situations like cooking where this would be amazing ("read the next step please, my hands are covered in raw pork", "how much flour for the roux", "crap, I don't have any lemons, what can I substitute")
Seems like a big win for language learning, if nothing else. Also seems possible to run locally, especially once the unsloth guys get their hands on it.
The qwen thinker/speaker architecture is really fascinating and is more in line with how I imagine human multi modality works - IE, a picture of an apple, the text a p p l e and the sound all map to the same concept without going to text first.
Isn’t that how all LLMs work?
The existing vision LLMs all work like this, which is most of the major models these days.
Multi-modal audio models are a lot less common. GPT-4o was meant to be able to do this natively from the start but they ended up shipping separate custom models based on it for their audio features. As far as I can tell GPT-5 doesn't have audio input/output at all - the OpenAI features for that still use GPT-4o-audio.
I don't know if Gemini 2.5 (which is multi-modal for vision and audio) shares the same embedding space for all three, but I expect it probably does.
What I mean is that all processing in an LLM occurs in state space. The next-token prediction is the very last step.
There are many more weird and complex architectures in models for video understanding.
For example, beyond video->text->llm and video->embedding in llm, you can also have an llm controlling/guiding a separate video extractor.
See this paper for a pretty thorough overview.
Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., Vosoughi, A., Huang, C., Zhang, Z., Liu, P., Feng, M., Zheng, F., Zhang, J., Luo, P., Luo, J., & Xu, C. (2025). Video Understanding with Large Language Models: A Survey (No. arXiv:2312.17432). arXiv. https://doi.org/10.48550/arXiv.2312.17432
Sure but all of these find some way of mapping inputs (any medium) to state space concepts. That's the core of the transformer architecture.
The user you originally replied to specifically mentioned > without going to text first
Yeah, and that's my understanding. Nothing goes video -> text, or audio -> text, or even text -> text without first going through state space. That's where the core of the transformer architecture is.
I'm running Q3-Next on my MBP and seeing ~ GPT4.1 performance from it.
Impressive what these local models are now capable of.
I recently needed to scan hundreds of low quality invoices and run them through OCR for invoice numbers and dates. I really took for granted how seamless this is in some applications, and was shocked how much work went into producing decent results.
I was obviously really naive. Either way, it gets me excited any time I see progress with OCR. I should give this a try against my (small) dataset.
I don't think I understand your comment. What were your results for Qwen? Or is that what you meant for how much work was needed?
What is your point about Qwen? Or is it just a general statement regarding LLM?
I did it the old school way with openai as the fallback.
All I'm saying is I'm excited to try Qwen to see if it out performs my gnarly algorithm.
Next steps for AI in general:
Does anyone have good resources for learning about multimodal models? I'm not sure where to begin.
https://x.com/whowillrickwill/status/1920723985311903767
I usually ask these models to tell me a short story, and most times the prose is stiff and the story reads like a mass market straight to KDP kids book. But wow, first shot generated something, light, mildly funny, and chill. Quite a surprise.
Pasted here for your own judgement:
*Title: The Last Lightbulb*
The power had been out for three days. Rain drummed against the windows of the old cabin, and the only light came from a flickering candle on the kitchen table.
Maggie, wrapped in a wool blanket, squinted at the last working flashlight. “We’ve got one bulb left, Jack. One.”
Jack, hunched over a board game he’d dug out of the closet, didn’t look up. “Then don’t turn it on unless you’re reading Shakespeare or delivering a baby.”
She rolled her eyes. “I need it to find the can opener. I’m not eating cold beans with my fingers again.”
Jack finally glanced up, grinning. “You did that yesterday and called it ‘rustic dining.’”
“Desperate times,” she muttered, clicking the flashlight on. The beam cut through the gloom—and immediately began to dim.
“No—!” Jack lunged, but too late. The light sputtered… then died.
Silence. Then Maggie sighed. “Well. There goes civilization.”
Jack leaned back, chuckling. “Guess we’re officially cavemen now.”
“Cavewoman,” she corrected, fumbling in the dark. “And I’m going to bed. Wake me when the grid remembers we exist.”
As she shuffled off, Jack called after her, “Hey—if you find the can opener in the dark, you’re officially magic.”
A pause. Then, from down the hall: “I found socks that match. That’s basically witchcraft.”
Jack smiled into the dark. “Goodnight, witch.”
“Goodnight, caveman.”
Outside, the rain kept falling. Inside, the dark didn’t feel so heavy anymore.
— The End —
will be interesting how it will compare with pricing with audio modality comparing to gemini 2.0 flash once many providers offer it.
Even though gemini 2.0 flash is quite old I still like it. Very cheap (each second of audio is just 32 tokens), support even more languages, non-reasoning so very fast, big rate limits.
Does this support realtime speech to speech via API? If so, where is this hosted/documented? I wasn’t able to see any info. I’d love to use this in lieu of OAIs (expensive) real time speech to speech offering.
https://www.alibabacloud.com/help/en/model-studio/realtime?s...
The multilingual example in the launch graphic has Qwen3 producing the text:
> "Bonjour, pourriez-vous me dire comment se rendreà la place Tian'anmen?"
translation: "Hello, could you tell me how to get to Tiananmen Square?"
a bold choice!
Westerners only know it from the massacre but it’s actually just like Times Square for them
Not really, it's a significant place which is why the protest (and hence massacre) was there, so especially for Chinese people (I expect) merely referencing it doesn't so immediately refer to the massacre, they have plenty of other connotations for it.
e.g. if something similar happened in Trafalgar Square, I expect it would still be primarily a major square in London to me, not oh my god they must be referring to that awful event. (In fact I think it was targeted in the 7/7 bombings for example.)
Or a better example to go with your translation - you can refer to the Bastille without 'boldly' invoking the histoire of its storming in the French Revolution.
No doubt the US media has referred to the Capitol without boldness many times since 6 Jan '21.
Not to mention, Tiananmen Square is one of the major tourist destinations in Beijing (similar to National Mall in Washington DC), for both domestic and foreign visitors.
This is true. I also think they've put some real effort into steering the model away from certain topics. If you ask too closely you'll get a response like:
"As an AI assistant, I must remind you that your statements may involve false and potentially illegal information. Please observe the relevant laws and regulations and ask questions in a civilized manner when you speak."
only really a reference with the date or at least 89