I find once there is one poisoned line of text it negatively affects everything else downstream. Instead, I use a temp/ folder with documents and use different files for different agents and models. Then I have to constantly prune and delete the files. Any information that can be extrapolated is just noise which negatively affects the agent. If you have a definition of a database structure and it has been implemented, that information should not be contained in any text document -- it is noise, will drift, and be impossible to debug why the agent keeps producing undesired behavior.
I have a ~/Projects folder. For example, I use Playwright with Chrome DevTools Protocol in order to do performance testing and leak detection. There is a script that handles this. My prompt is "Search ~/Projects for perf testing with CDP and Playwright and implement here". Point being, if I need anything I point to a resource or ask to search a resource and it will find it quick and, most importantly, tends to improve it every iteration.
If I was in an institution, I would have a repository and would rather just point the resource and say use that than have memory of it locally.
Memory is prone to poisoning. Also, I want to be able to take my toys and go elsewhere.
My method is seven layers of files, administered differently:
1. meta-knowledge,
2. project seed,
3. LLM wiki,
4. code,
5. tickets and todos,
6. chat logs,
7. artifactory.
Ordered idea-to-reality. All git repos.
That outgrows any context window pretty soon. My way out of that trap is to use links, both wiki links and git permalinks.
What I propose is only a very slightly more formal version of what you describe.
Just to start with: memoryfields are possible to use in a server/client system. That was a key aim and I do already use them over Amazon S3 (though not always).
I started, like you did, with a personal library of prompts. But the issue is that as your library of little pieces of prompts increases a) you get tired of constantly editing them yourself b) you have no easy way to export and share them with others c) it's frustrating that the agent doesn't "automatically" find your little bit of prompt on X even when clearly it is relevant - hence sem search.
I think a lot of people are still using the "personal library of bits of prompt" model. It is ok. But I wanted to propose an minimal, interchangeable standard for sharing them. So the idea of being an institution and having a shared memoryfield: that's something I want as well!
Your experience mirrors my own. I don't know if he coined the term, but Steve Yegge talks about 'heresies' that creep in to a system -- untrue things that stick around and permanently influence its behavior. I still find that these happen regularly and stopped using self-managing memory systems because they make heresies even harder to diagnose and remove.
Within projects, I make heavy use of path-scoped rules to intentionally bring context where it's needed, and also make heavy use of temp directories. LLMs are more than happy to produce ad-hoc memories/summaries/context docs that I can then point a session to, but I can be selective and intentional about it.
I like that memoryfield is portable, intentional and composable. I'm not convinced that sharing memoryfields between users will be practical, but I keep isolated virtual environments for absolutely everything. I like the idea of being able to intentionally bring collections of managed context around with me. There are other ways to do that, but will keep an eye on this.
This is a real problem, and it’s not just in markdown files and docs. Claude loves to write things we’ve “discovered” in comments and then later in treats the comments as gospel truth.
You have to constantly tend the garden and weed these things out.
Whenever Claude makes some error ask it where and why? And then dig out the weed.
And of course if it’s in the context, probably time for a handover doc (which will need weeding) and started fresh.
I have mine self record into Supabase for my projects and a local sqlite for work. It determines a method of record keeping which I audit every week to hone the process. Really makes it so I can move to any provider I want and I have a queryable memory store. Also really helps when someone asks about why some feature was implemented a certain way. I also have it learn from corrections in PRs and comments made overtime in a repo to get the shape of what is important to the team at work.
I keep it on for my web chats, but I think I dislike it more than it helps. I'll ask a question about something and it'll find a way to tie it back to something from three months ago thinking it was a deep project I was working on, instead of what it really was: an inane question I was curious about.
I turn it off for local agents because I bounce around a few and I really don't like the mostly implicit nature of it. I want to write my instructions in version control if I have anything to say consistently to an agent.
"Irrelevant material is simply never surfaced by the semantic search."
thats quite optimistic. there's lots of "memory" or past chats with agents that should be suppressed and forgotten because they were looking in the wrong place or were eventually proven wrong. yet semantically they'd look very relevant to a future search. thats why you shouldn't search both textbooks and scifi when trying to solve an examination.
Was ready to write something snarky because this is essentially RAG, but I think the author is getting at some subtle details which are seemingly important.
- memory systems are a specific type of knowledge base where you generate all the documents. You might as well generate them to be less than your embedding token limit to obviate the need for chunking.
- embedding models are getting better and are no longer just semantic averaging.
- small models are getting dirt cheap, making parallel reads cost manageable
What they describe is sort of the simplest architecture that takes advantage of these observations. I believe them when they say it works well.
I do suspect though that things like keyword lookup will completely fail if every memory is just a vector. Hence why something like Typesense hybrid search can still be useful.
It occurs to me: we have latent embedding giving 'general knowledge' to an LLM. What if we use a 'blank' LLM as well as an agent and train that blank LLM on personal context to query that as memory?
I'm starting to think that 'memory' may be the wrong analogy for what we want.
I do think that having a set of token that are highly personalized to your project and to way you work is beneficial. I also think that the idea that this set of token will be constructed in the background without any work from the user is really appealing. So it's understandable that the 'memory' analogy became so popular.
But in my experience having a really good AGENTS.md file almost always produce better results than enabling memory.
Maybe we should start to think about how we 'train'/'onboard' agents into our projects, in a similar way that we do for new co-workers. Imagine if we could send the agent to our repo and ask it to learn our patterns and in the end we could quiz the agent to gauge how much it actually understood the project. Once he 'understands' the project we can start to use it to help with development.
In a very small scale (example, individual new features) I will sometimes ask the agent to explain me how things work (even though I already know how it works) so I can 'prime' the agent context with good data before starting any real work. But I'm not sure if this approach could be reliably scaled to work with any repo for any kind of work.
When I was doing a really large refactor across the codebase I told Claude Code to explain how certain things worked currently and how I wanted things to look like after the migration and a plan on how to get there. Then I did several further clean sessions where I always started along the lines: "Using this plan {link} do X."
Works pretty well for "non-permanent" instructions (you don't want to put this info into your committed markdowns).
My main motivation was simply to save tokens, but it actually worked really well and improved speed as well.
I just had a thought about similar thing - how to track human decisions on the codebase? Consider you are writing code together with AI, how you understand which code change happened because human asked for it
This was a compelling writeup to me. I read through the spec and found it easy to understand and make sense of.
I wonder how much my system needs something like this. Between the invisible system memory of my random chats with Gippity, my Matt Pocock skills saving terminology and plans, and whatever else Cursor and Codex do, I don't think I feel a need for more agent memory. I do like how it's exposed and searchable, and not invisible. But I honestly just send my questions/tasks away to my magic agent and eventually it gets it right anyway; do I need more discrete memory my team has to maintain? (That's an earnest question, not disregard for this)
What the author suggests is remarkably close to the proposition of OpenViking. I've been testing a few memory solutions and OpenViking is one of my favorites so far.
It’s good that a lot of people are trying a lot of things when it comes to agentic memory. Sadly none of it represents a complete solution at this time. But we need the experimentation.
I still dont understand what "agentic memory" is . agents can already call sql / rag and grep through files or whatever. why is "agentic memory" a special thing.
Agents have to decide to search files, and they don't always know that they should. For example, if the agent sees the database, it may miss detailed instructions on how to access the database deeper in the repo. Behaviorally, humans want to be able to say "hey, here's how to access the DB. Remember that." Or better yet, for it to happen automatically.
yea ofcourse you have tell agent how to access you database. But why does it have a specail name, what the big deal about giving agent info how to call your db.
You could think of it as a more efficient indexing format for data the agent has access to. It’s badly needed as a better representation or pointer map would improve recall time and comprehensiveness dramatically without having to invest further in model training.
Memory isn’t the same thing as rag it’s usually just a text based index of past events and the llm reads it and decides what’s important rather than querying a db
they just want to reinvent information retrieval from first principles; also the fact that everyone forgets to check what currently exists and reinvents hexagonal wheels for the sake of agentic development
I've come to a similar lofi solution for my agent fleet. Markdown wiki with simple querying is decently effective as a memory system. Setting up a skill that can effectively reduce a session into useful long term lessons is the easiest unlock for these systems.
If you look at how agents navigate source code, they do not look at directory names, and drill down into the ones with plausible names, instead the grep the whole repo for plausible keywords.
Of course, ideally your data would be structured, but the agents will mostly be grepping anyway, and maybe look at sibbling files.
I see a lot of claims in this article without ... any proof?
Both can be true:
- It's useful to anthropomorphize agents when predicting behavior and
- we have to use specific language to specify what we mean.
What does the author mean by "confuse the models" ? Are they talking about not picking right information? Picking the wrong information? Losing their previous context / task?
Part of setting up a proper eval is also deciding what we actually mean ourself. What are we actually optimizing for? It's not, e.g. % confusion, %rubbish, etc.
The article does point to it: retrieval latency, accuracy, etc.
that sounds horribly token inefficient, just create a tool call if you are all in on the agentic approach and hide memory retrieval behind an optimized api
They already have. The tool is called bash and optimized api is grep, the storage is the file system. We’ve been here many months.
What people are exploring are other options as far as I can tell.
What you are offering is “don’t do that, this already works”. Which I guess is fine, but apparently not everyone is fully satisfied with the current generation of tooling.
Yeah but probably the whole of the memories for any given project will likely fit into context, and the agent can’t extract whatever it wants for the given task. Grep only comes in if they choose to narrow down the candidate memory files, and they are usually searching many keywords and synonyms.
Of course a real search system with ranking and whatnot would be better, but you’re paying a different cost there.
I’m personally more interested in how the memory files get created and updated and generally managed, retrieving the correct ones doesn’t seem to be a big problem at the moment.
I primarily use gh copilot and they are by default hidden from the user. It’s also not great that they aren’t in the repo and every contributor has their own set of memories of different freshness, likely conflicting.
I find that semantic search is substantially better than keyword search even for small corpuses. Being able to find "related" material that doesn't match the keyword is a big advance over traditional full text search.
> How can I judge what is a good memory to store? How can I avoid filling my memory with crap?
> This is a common fear with memory systems but doesn't really apply to memoryfields. Irrelevant material is simply never surfaced by the semantic search.
This is so wrong. The Achilles' heel of this approach is the RAG. What makes it worse is having lots of memories that are outdated, wrong, hallucinated, or irrelevant.
Nothing beats curated data. Memory should be regularly reviewed, compacted, and cleaned up if it's no longer valid.
Does anyone else not use memory?
I find once there is one poisoned line of text it negatively affects everything else downstream. Instead, I use a temp/ folder with documents and use different files for different agents and models. Then I have to constantly prune and delete the files. Any information that can be extrapolated is just noise which negatively affects the agent. If you have a definition of a database structure and it has been implemented, that information should not be contained in any text document -- it is noise, will drift, and be impossible to debug why the agent keeps producing undesired behavior.
I have a ~/Projects folder. For example, I use Playwright with Chrome DevTools Protocol in order to do performance testing and leak detection. There is a script that handles this. My prompt is "Search ~/Projects for perf testing with CDP and Playwright and implement here". Point being, if I need anything I point to a resource or ask to search a resource and it will find it quick and, most importantly, tends to improve it every iteration.
If I was in an institution, I would have a repository and would rather just point the resource and say use that than have memory of it locally.
Memory is prone to poisoning. Also, I want to be able to take my toys and go elsewhere.
My method is seven layers of files, administered differently:
1. meta-knowledge,
2. project seed,
3. LLM wiki,
4. code,
5. tickets and todos,
6. chat logs,
7. artifactory.
Ordered idea-to-reality. All git repos. That outgrows any context window pretty soon. My way out of that trap is to use links, both wiki links and git permalinks.
http://replicated.live/blog/wiki
What I propose is only a very slightly more formal version of what you describe.
Just to start with: memoryfields are possible to use in a server/client system. That was a key aim and I do already use them over Amazon S3 (though not always).
I started, like you did, with a personal library of prompts. But the issue is that as your library of little pieces of prompts increases a) you get tired of constantly editing them yourself b) you have no easy way to export and share them with others c) it's frustrating that the agent doesn't "automatically" find your little bit of prompt on X even when clearly it is relevant - hence sem search.
I think a lot of people are still using the "personal library of bits of prompt" model. It is ok. But I wanted to propose an minimal, interchangeable standard for sharing them. So the idea of being an institution and having a shared memoryfield: that's something I want as well!
The spec, feedback greatly welcome:
https://github.com/calpaterson/memoryfield-spec/blob/main/SP...
Your experience mirrors my own. I don't know if he coined the term, but Steve Yegge talks about 'heresies' that creep in to a system -- untrue things that stick around and permanently influence its behavior. I still find that these happen regularly and stopped using self-managing memory systems because they make heresies even harder to diagnose and remove.
Within projects, I make heavy use of path-scoped rules to intentionally bring context where it's needed, and also make heavy use of temp directories. LLMs are more than happy to produce ad-hoc memories/summaries/context docs that I can then point a session to, but I can be selective and intentional about it.
I like that memoryfield is portable, intentional and composable. I'm not convinced that sharing memoryfields between users will be practical, but I keep isolated virtual environments for absolutely everything. I like the idea of being able to intentionally bring collections of managed context around with me. There are other ways to do that, but will keep an eye on this.
This is a real problem, and it’s not just in markdown files and docs. Claude loves to write things we’ve “discovered” in comments and then later in treats the comments as gospel truth.
You have to constantly tend the garden and weed these things out.
Whenever Claude makes some error ask it where and why? And then dig out the weed.
And of course if it’s in the context, probably time for a handover doc (which will need weeding) and started fresh.
I have mine self record into Supabase for my projects and a local sqlite for work. It determines a method of record keeping which I audit every week to hone the process. Really makes it so I can move to any provider I want and I have a queryable memory store. Also really helps when someone asks about why some feature was implemented a certain way. I also have it learn from corrections in PRs and comments made overtime in a repo to get the shape of what is important to the team at work.
I keep it on for my web chats, but I think I dislike it more than it helps. I'll ask a question about something and it'll find a way to tie it back to something from three months ago thinking it was a deep project I was working on, instead of what it really was: an inane question I was curious about.
I turn it off for local agents because I bounce around a few and I really don't like the mostly implicit nature of it. I want to write my instructions in version control if I have anything to say consistently to an agent.
"Irrelevant material is simply never surfaced by the semantic search." thats quite optimistic. there's lots of "memory" or past chats with agents that should be suppressed and forgotten because they were looking in the wrong place or were eventually proven wrong. yet semantically they'd look very relevant to a future search. thats why you shouldn't search both textbooks and scifi when trying to solve an examination.
Was ready to write something snarky because this is essentially RAG, but I think the author is getting at some subtle details which are seemingly important.
- memory systems are a specific type of knowledge base where you generate all the documents. You might as well generate them to be less than your embedding token limit to obviate the need for chunking.
- embedding models are getting better and are no longer just semantic averaging.
- small models are getting dirt cheap, making parallel reads cost manageable
What they describe is sort of the simplest architecture that takes advantage of these observations. I believe them when they say it works well.
I do suspect though that things like keyword lookup will completely fail if every memory is just a vector. Hence why something like Typesense hybrid search can still be useful.
It occurs to me: we have latent embedding giving 'general knowledge' to an LLM. What if we use a 'blank' LLM as well as an agent and train that blank LLM on personal context to query that as memory?
That's a whole lot of text to say "it's markdown".
It feels like 50% of AI "progress" is just finding different ways to say "tell the model things in English."
That's only half. It's a while lot of text to say "it's markdown + RAG/semantic search".
The markdown explanation I was also unimpressed with, but RAG over hyperlinks is convincing to me.
A whole lot of text to say "Just use text".
I'm starting to think that 'memory' may be the wrong analogy for what we want.
I do think that having a set of token that are highly personalized to your project and to way you work is beneficial. I also think that the idea that this set of token will be constructed in the background without any work from the user is really appealing. So it's understandable that the 'memory' analogy became so popular.
But in my experience having a really good AGENTS.md file almost always produce better results than enabling memory.
Maybe we should start to think about how we 'train'/'onboard' agents into our projects, in a similar way that we do for new co-workers. Imagine if we could send the agent to our repo and ask it to learn our patterns and in the end we could quiz the agent to gauge how much it actually understood the project. Once he 'understands' the project we can start to use it to help with development.
In a very small scale (example, individual new features) I will sometimes ask the agent to explain me how things work (even though I already know how it works) so I can 'prime' the agent context with good data before starting any real work. But I'm not sure if this approach could be reliably scaled to work with any repo for any kind of work.
When I was doing a really large refactor across the codebase I told Claude Code to explain how certain things worked currently and how I wanted things to look like after the migration and a plan on how to get there. Then I did several further clean sessions where I always started along the lines: "Using this plan {link} do X."
Works pretty well for "non-permanent" instructions (you don't want to put this info into your committed markdowns).
My main motivation was simply to save tokens, but it actually worked really well and improved speed as well.
It could also be shareable?
I just had a thought about similar thing - how to track human decisions on the codebase? Consider you are writing code together with AI, how you understand which code change happened because human asked for it
This was a compelling writeup to me. I read through the spec and found it easy to understand and make sense of.
I wonder how much my system needs something like this. Between the invisible system memory of my random chats with Gippity, my Matt Pocock skills saving terminology and plans, and whatever else Cursor and Codex do, I don't think I feel a need for more agent memory. I do like how it's exposed and searchable, and not invisible. But I honestly just send my questions/tasks away to my magic agent and eventually it gets it right anyway; do I need more discrete memory my team has to maintain? (That's an earnest question, not disregard for this)
What the author suggests is remarkably close to the proposition of OpenViking. I've been testing a few memory solutions and OpenViking is one of my favorites so far.
It’s good that a lot of people are trying a lot of things when it comes to agentic memory. Sadly none of it represents a complete solution at this time. But we need the experimentation.
I still dont understand what "agentic memory" is . agents can already call sql / rag and grep through files or whatever. why is "agentic memory" a special thing.
Agents have to decide to search files, and they don't always know that they should. For example, if the agent sees the database, it may miss detailed instructions on how to access the database deeper in the repo. Behaviorally, humans want to be able to say "hey, here's how to access the DB. Remember that." Or better yet, for it to happen automatically.
That's agentic memory.
yea ofcourse you have tell agent how to access you database. But why does it have a specail name, what the big deal about giving agent info how to call your db.
Just retrieving usually doesn't count as memory. "Memory" tends to imply writing too.
And I agree: it's not a very special thing. That's why I propose: Markdown + a simple embedding.
You could think of it as a more efficient indexing format for data the agent has access to. It’s badly needed as a better representation or pointer map would improve recall time and comprehensiveness dramatically without having to invest further in model training.
Memory isn’t the same thing as rag it’s usually just a text based index of past events and the llm reads it and decides what’s important rather than querying a db
Sounds like RAG to me.
they just want to reinvent information retrieval from first principles; also the fact that everyone forgets to check what currently exists and reinvents hexagonal wheels for the sake of agentic development
Right now you have an opportunity to enlighten everyone rather than talk down to them.
Wouldn't Avro or Parquet be solid choices for something like this, or am I out of date & out of touch?
Memory is not just a matter of retrieval, it's also a matter of knowing what to retrieve and when.
I've come to a similar lofi solution for my agent fleet. Markdown wiki with simple querying is decently effective as a memory system. Setting up a skill that can effectively reduce a session into useful long term lessons is the easiest unlock for these systems.
I'm not convinced an unstructured collection of memory files is the way to go at all.
If you look at how agents navigate source code, they do not look at directory names, and drill down into the ones with plausible names, instead the grep the whole repo for plausible keywords.
Of course, ideally your data would be structured, but the agents will mostly be grepping anyway, and maybe look at sibbling files.
its the same way they crawl websites, its horribly inefficient
I see a lot of claims in this article without ... any proof?
Both can be true: - It's useful to anthropomorphize agents when predicting behavior and - we have to use specific language to specify what we mean.
What does the author mean by "confuse the models" ? Are they talking about not picking right information? Picking the wrong information? Losing their previous context / task?
Part of setting up a proper eval is also deciding what we actually mean ourself. What are we actually optimizing for? It's not, e.g. % confusion, %rubbish, etc.
The article does point to it: retrieval latency, accuracy, etc.
Agent memory is to computer memory is what Mongo DB is to relational database.
Incredible to watch things come full circle. Next thing you know, someone is going to figure out a binary encoding.
agents can call relational databases fine. infact, really good at sql if you can store your memories into that format.
that sounds horribly token inefficient, just create a tool call if you are all in on the agentic approach and hide memory retrieval behind an optimized api
They already have. The tool is called bash and optimized api is grep, the storage is the file system. We’ve been here many months.
What people are exploring are other options as far as I can tell.
What you are offering is “don’t do that, this already works”. Which I guess is fine, but apparently not everyone is fully satisfied with the current generation of tooling.
I was responding specifically to the parent comment that was suggesting generating ad-hoc sql.
Also fwiw grep is pretty poorly suited to semantic search and will only return the most basic of matches.
If you are really trying to build a useful memory search tool there are much better options than plain text search.
Yeah but probably the whole of the memories for any given project will likely fit into context, and the agent can’t extract whatever it wants for the given task. Grep only comes in if they choose to narrow down the candidate memory files, and they are usually searching many keywords and synonyms.
Of course a real search system with ranking and whatnot would be better, but you’re paying a different cost there.
I’m personally more interested in how the memory files get created and updated and generally managed, retrieving the correct ones doesn’t seem to be a big problem at the moment.
I primarily use gh copilot and they are by default hidden from the user. It’s also not great that they aren’t in the repo and every contributor has their own set of memories of different freshness, likely conflicting.
Are embeddings useful for something of the scale compared to just keyword search (aka grep)?
I find that semantic search is substantially better than keyword search even for small corpuses. Being able to find "related" material that doesn't match the keyword is a big advance over traditional full text search.
How well does it work in practice?
the new OpenAI spec is agent memory as file names
Which spec are you referring to?
the IM1 HuggingFace spec
> How can I judge what is a good memory to store? How can I avoid filling my memory with crap?
> This is a common fear with memory systems but doesn't really apply to memoryfields. Irrelevant material is simply never surfaced by the semantic search.
This is so wrong. The Achilles' heel of this approach is the RAG. What makes it worse is having lots of memories that are outdated, wrong, hallucinated, or irrelevant.
Nothing beats curated data. Memory should be regularly reviewed, compacted, and cleaned up if it's no longer valid.