I've been documenting my house in software for years: outlets, conduits, circuits, pipe runs, appliances, vehicles. Having that data came in handy more often than I expected. But adding data was always the bottleneck. Opening the app, navigating to the right room, picking categories, filling in fields.
So I built an MCP server for that database. Now Claude, Codex, and other MCP-compatible clients can read and write the home data directly. Tell it about a heat pump in one sentence, or snap a photo of the nameplate, and it figures out the category and location. Later you can ask whether it's worth repairing given its age.
It also tracks physical connections between elements: cables, pipes, ducts, and where they run. The database is a single local file on your machine.
Yeah this is a killer app for mcp servers. My puppy has food allergies and the vet asked me to track his eating and pooping. I ended up building an mcp server to do this data entry and now I track his activity and planning to track also his training progress. It's very different when you can just tell your model "he ate 2 cups of his kibble" or "we practiced stay for 5 minutes" vs doing tedious data entry. As a bonus this helps my partner and me coordinate dog care so we can have fewer conversations about the dog
Interesting. I use HA myself (though not the AI/LLM side yet). I think it's a different layer: Home Memory is the physical reality of the whole house, not just what's wired and smart. How do you picture "running inside"? Chat from the HA UI, or just running on the same hardware? I haven't dug into the feasibility yet. The part I'm fairly sure about: this only shines with a model that reliably uses tools. 23 MCP tools is a lot for a small local model, Claude or GPT tier handles it fine. What conversation agent are you running in HA?
I've been running Home Assistant for 5 years now. No turning back, it's so addictive (in a good sense). I didn't have a chance to start with AI/LLM in HA either, nor did I ever have a chance to use speech with HA. It will all come in this year, I hope. The other day, going through a cluttered drawer, looking for something, it dawned on me that the perfect solution to that problem would be if I could talk to an AI and explain that item X is in drawer Y of cabinet Z in room A. That would be a perfect interface to an inventory management application. And, since I'm using HA more and more for absolutely everything in my life, then, the perfect place to keep all that data would be HA itself. Later on, all that data will be useful in HA anyway, one way or the other. So, yes, if I need to build talk-to-LLM from every room infrastructure, then it would make it much harder to include other projects in the pipeline. HA mobile app already has talk-to-HA functionality which I can't use for Home Memory. And, frankly, having some home related data in Home Assistant, and some other home related data in Home Memory feels like a split brain situation to me. So, ideally, Home Memory should be just an integration/add-on/HACS to Home Assistant, in my humble opinion. What you did there with a windows server is a great start, and I will definitely test it out, but, eventually, I strongly believe it should be fully integrated into Home Assistant. Let me know if I can be of help. Thanks.
You just gave me the idea to 1) photo a cluttered drawer and have AI identify all the things, 2) it'd be nice if it could take that output and structure it (eg, items A, B, and C are in drawer 1) and 3) maybe even have a cheap cameras in the important places that update periodically. I dunno. Also maybe have it connect to this home memory system, ill check it out more. Thank you both!
I’m curious how different is it compared to use an AI agent to get your transcript, summarize and document it a file, and then ask it something? Like a simple RAG or even just project in codex
For a small set of notes, an agent over files works fine. The difference here is that the house is stored as rooms, devices, cables, connections, and dates, not just as summaries. That makes questions like "which cables leave this panel and where do they end?" or "which appliances in the utility room are older than 10 years?" much more reliable.
With files or transcripts, the model has to reconstruct that structure from prose every time, and a fully documented house quickly runs into context-window limits. Here it makes small, focused queries that return just the slice it needs.
For a small set of notes, an agent over files works fine. The difference here is that the house is stored as rooms, devices, cables, connections, and dates, not just as summaries. That makes questions like "which cables leave this panel and where do they end?" or "which appliances in the utility room are older than 10 years?" much more reliable.
With files or transcripts, the model has to reconstruct that structure from prose every time, and a fully documented house quickly runs into context-window limits. Here it makes small, focused queries that return just the slice it needs.
Late here in Austria. Happy to follow up tomorrow.
Good association. I actually considered "Home Twin" early on, but moved away from it because "digital twin" suggests real-time sync, telemetry, or simulation. This is closer to a memory of the house: what's there, how it connects, and what happened to it over time. "Home Memory" felt like a better fit.
Still, images and even models could have some merit. Imagine handing this off to an elderly parent who insists they can figure out everything by themselves. Or being able to measure how to run wiring to a new switch (in my century home, the wiring is spaghetti behind the walls and ceiling)
Images definitely have their place. The desktop app this grew out of lets me attach photos to elements. But even simple structured positions and routes cover a lot of day-to-day cases: "switch at 110 cm, 15 cm from the left edge", cable runs section by section, etc.
For true old-house spaghetti behind walls, photos or sketches probably matter more. But for handoff and routine lookups, being able to ask for all switches in an area and their positions beats digging through scattered notes.
I've been documenting my house in software for years: outlets, conduits, circuits, pipe runs, appliances, vehicles. Having that data came in handy more often than I expected. But adding data was always the bottleneck. Opening the app, navigating to the right room, picking categories, filling in fields.
So I built an MCP server for that database. Now Claude, Codex, and other MCP-compatible clients can read and write the home data directly. Tell it about a heat pump in one sentence, or snap a photo of the nameplate, and it figures out the category and location. Later you can ask whether it's worth repairing given its age.
It also tracks physical connections between elements: cables, pipes, ducts, and where they run. The database is a single local file on your machine.
Project site with demo video: https://home-memory.com
Yeah this is a killer app for mcp servers. My puppy has food allergies and the vet asked me to track his eating and pooping. I ended up building an mcp server to do this data entry and now I track his activity and planning to track also his training progress. It's very different when you can just tell your model "he ate 2 cups of his kibble" or "we practiced stay for 5 minutes" vs doing tedious data entry. As a bonus this helps my partner and me coordinate dog care so we can have fewer conversations about the dog
This would actually be the best to be running inside Home Assistant.
Interesting. I use HA myself (though not the AI/LLM side yet). I think it's a different layer: Home Memory is the physical reality of the whole house, not just what's wired and smart. How do you picture "running inside"? Chat from the HA UI, or just running on the same hardware? I haven't dug into the feasibility yet. The part I'm fairly sure about: this only shines with a model that reliably uses tools. 23 MCP tools is a lot for a small local model, Claude or GPT tier handles it fine. What conversation agent are you running in HA?
I've been running Home Assistant for 5 years now. No turning back, it's so addictive (in a good sense). I didn't have a chance to start with AI/LLM in HA either, nor did I ever have a chance to use speech with HA. It will all come in this year, I hope. The other day, going through a cluttered drawer, looking for something, it dawned on me that the perfect solution to that problem would be if I could talk to an AI and explain that item X is in drawer Y of cabinet Z in room A. That would be a perfect interface to an inventory management application. And, since I'm using HA more and more for absolutely everything in my life, then, the perfect place to keep all that data would be HA itself. Later on, all that data will be useful in HA anyway, one way or the other. So, yes, if I need to build talk-to-LLM from every room infrastructure, then it would make it much harder to include other projects in the pipeline. HA mobile app already has talk-to-HA functionality which I can't use for Home Memory. And, frankly, having some home related data in Home Assistant, and some other home related data in Home Memory feels like a split brain situation to me. So, ideally, Home Memory should be just an integration/add-on/HACS to Home Assistant, in my humble opinion. What you did there with a windows server is a great start, and I will definitely test it out, but, eventually, I strongly believe it should be fully integrated into Home Assistant. Let me know if I can be of help. Thanks.
You just gave me the idea to 1) photo a cluttered drawer and have AI identify all the things, 2) it'd be nice if it could take that output and structure it (eg, items A, B, and C are in drawer 1) and 3) maybe even have a cheap cameras in the important places that update periodically. I dunno. Also maybe have it connect to this home memory system, ill check it out more. Thank you both!
I’m curious how different is it compared to use an AI agent to get your transcript, summarize and document it a file, and then ask it something? Like a simple RAG or even just project in codex
For a small set of notes, an agent over files works fine. The difference here is that the house is stored as rooms, devices, cables, connections, and dates, not just as summaries. That makes questions like "which cables leave this panel and where do they end?" or "which appliances in the utility room are older than 10 years?" much more reliable. With files or transcripts, the model has to reconstruct that structure from prose every time, and a fully documented house quickly runs into context-window limits. Here it makes small, focused queries that return just the slice it needs.
For a small set of notes, an agent over files works fine. The difference here is that the house is stored as rooms, devices, cables, connections, and dates, not just as summaries. That makes questions like "which cables leave this panel and where do they end?" or "which appliances in the utility room are older than 10 years?" much more reliable.
With files or transcripts, the model has to reconstruct that structure from prose every time, and a fully documented house quickly runs into context-window limits. Here it makes small, focused queries that return just the slice it needs.
Late here in Austria. Happy to follow up tomorrow.
Neat idea to do this with a house! https://en.wikipedia.org/wiki/Digital_twin
Good association. I actually considered "Home Twin" early on, but moved away from it because "digital twin" suggests real-time sync, telemetry, or simulation. This is closer to a memory of the house: what's there, how it connects, and what happened to it over time. "Home Memory" felt like a better fit.
Still, images and even models could have some merit. Imagine handing this off to an elderly parent who insists they can figure out everything by themselves. Or being able to measure how to run wiring to a new switch (in my century home, the wiring is spaghetti behind the walls and ceiling)
Images definitely have their place. The desktop app this grew out of lets me attach photos to elements. But even simple structured positions and routes cover a lot of day-to-day cases: "switch at 110 cm, 15 cm from the left edge", cable runs section by section, etc.
For true old-house spaghetti behind walls, photos or sketches probably matter more. But for handoff and routine lookups, being able to ask for all switches in an area and their positions beats digging through scattered notes.