Hi Jacob, thanks for sharing! Witchcraft looks super interesting—starring the repo now.
The late interaction (ColBERT / MaxSim token-level) approach is fascinating for local search. Retaining per-token interaction matrices gives incredible semantic retrieval precision without losing fine-grained token context.
From an architectural standpoint, the main trade-off I explored with Hillock was moving away from dense token-embedding matrices entirely toward an explicit neuro-symbolic split:
1. Symbolic Grounding: Hard Subject-Predicate-Object (SPO) triples in relational SQLite tables to eliminate vector drift for exact factual memory.
2. Subsymbolic VSA Gating: A 10,000-dimensional Vector Symbolic Architecture (HDC) hypervector space on CPU using subword n-grams and GloVe SimHash projections for <1ms gating and pronoun resolution.
3. Edge VRAM Footprint: Keeping the whole engine under 1.2 GB VRAM on a GTX 1070 by using a CUDA bi-encoder pipeline (Fastcoref + MiniLM + GLiREL) for non-generative document parsing.
I'm curious—how do you manage the index footprint and per-token memory overhead in Witchcraft when scaling to larger local document collections?
I built Hillock (https://github.com/roandejager/Hillock), a 100% local, offline neuro-symbolic memory engine designed to replace heavy Vector DBs and token-hungry LLM document ingestion pipelines on edge hardware.
### The Problem with Local RAG
Standard local RAG is surprisingly heavy. Running dense vector databases alongside an 8B+ generative LLM just to parse documents and maintain conversational context burns VRAM, chokes mid-range GPUs (like a GTX 1070), and still hallucinates when asked about missing or unverified information.
### The Architecture
Instead of using autoregressive LLMs to generate JSON extractions, Hillock decouples memory into three lightweight tiers:
1. SQLite Knowledge Graph (database.py): Ground-truth Subject-Predicate-Object (SPO) triples stored in relational tables. Zero vector drift for factual data.
2. Hebbian Plasticity Engine (plasticity.py): Gradient-free co-activation associative learning that updates synaptic weights between entity nodes across chat turns.
3. 10,000-D VSA/HDC Reservoir (reservoir.py): A Vector Symbolic Architecture hypervector space using Subword N-Grams (3,4,5-grams) and Locality-Sensitive Sign Random Projections (SimHash) over a static 10MB GloVe dictionary. It handles pronoun resolution and context similarity gating on CPU in under 1 millisecond.
Ingestion completes in ~3.9–5.0 seconds for a 32-sentence academic document (6.3–8.0 sentences/sec pure GPU rate) using <1.2 GB VRAM on a GTX 1070. Generative LLMs (via local Ollama) are invoked ONLY for final response rendering once a query passes the HDC similarity gate.
### What's New in v0.4
In the v0.4 release series, we tackled zero-shot extraction noise and inverted facts:
- O(1) Type-Constrained Schema Validation: Set-based domain/codomain filtering over GLiREL outputs.
- Automatic Direction Correction: Detects and swaps inverted extractions (e.g., turning [Budapest born_in John von Neumann] into [John von Neumann born_in Budapest]).
- Precompiled Entity Sanitization: Strips possessive artifacts ('s), trailing action verbs, and prepositional tails.
- Raw extraction precision improved from 11.5% to 15.5% (+35% cleaner graph).
### Current Benchmark Status (Honest Baseline)
On a fixed 32-sentence benchmark (20 answerable queries, 10 hard-negative trick queries):
- Retrieval Accuracy: 55.0%
- Extraction Recall: 50.0%
- Extraction Precision: 15.5%
- Fast-Eval Retrieval Duration: ~1.27s for 30 queries (~0.04s per query)
- Unanswerable Queries: 0 GPU tokens wasted (CPU similarity gate shuts down LLM calls instantly).
Here is a project with similar goals (I am the main author), but using late interaction semantic search: https://github.com/dropbox/witchcraft
Hi Jacob, thanks for sharing! Witchcraft looks super interesting—starring the repo now.
The late interaction (ColBERT / MaxSim token-level) approach is fascinating for local search. Retaining per-token interaction matrices gives incredible semantic retrieval precision without losing fine-grained token context.
From an architectural standpoint, the main trade-off I explored with Hillock was moving away from dense token-embedding matrices entirely toward an explicit neuro-symbolic split:
1. Symbolic Grounding: Hard Subject-Predicate-Object (SPO) triples in relational SQLite tables to eliminate vector drift for exact factual memory. 2. Subsymbolic VSA Gating: A 10,000-dimensional Vector Symbolic Architecture (HDC) hypervector space on CPU using subword n-grams and GloVe SimHash projections for <1ms gating and pronoun resolution. 3. Edge VRAM Footprint: Keeping the whole engine under 1.2 GB VRAM on a GTX 1070 by using a CUDA bi-encoder pipeline (Fastcoref + MiniLM + GLiREL) for non-generative document parsing.
I'm curious—how do you manage the index footprint and per-token memory overhead in Witchcraft when scaling to larger local document collections?
Excited to dig deeper into your codebase!
Hi HN,
I built Hillock (https://github.com/roandejager/Hillock), a 100% local, offline neuro-symbolic memory engine designed to replace heavy Vector DBs and token-hungry LLM document ingestion pipelines on edge hardware.
### The Problem with Local RAG Standard local RAG is surprisingly heavy. Running dense vector databases alongside an 8B+ generative LLM just to parse documents and maintain conversational context burns VRAM, chokes mid-range GPUs (like a GTX 1070), and still hallucinates when asked about missing or unverified information.
### The Architecture Instead of using autoregressive LLMs to generate JSON extractions, Hillock decouples memory into three lightweight tiers:
1. SQLite Knowledge Graph (database.py): Ground-truth Subject-Predicate-Object (SPO) triples stored in relational tables. Zero vector drift for factual data. 2. Hebbian Plasticity Engine (plasticity.py): Gradient-free co-activation associative learning that updates synaptic weights between entity nodes across chat turns. 3. 10,000-D VSA/HDC Reservoir (reservoir.py): A Vector Symbolic Architecture hypervector space using Subword N-Grams (3,4,5-grams) and Locality-Sensitive Sign Random Projections (SimHash) over a static 10MB GloVe dictionary. It handles pronoun resolution and context similarity gating on CPU in under 1 millisecond.
### Document Ingestion (TALON Engine) Document parsing uses TALON—a 3-stage CUDA pipeline: - Stage 1: Document coreference preprocessing (Fastcoref) - Stage 2: Bi-Encoder candidate predicate routing (all-MiniLM-L6-v2 in <2ms) - Stage 3: Zero-shot span relation extraction (GLiREL Large)
Ingestion completes in ~3.9–5.0 seconds for a 32-sentence academic document (6.3–8.0 sentences/sec pure GPU rate) using <1.2 GB VRAM on a GTX 1070. Generative LLMs (via local Ollama) are invoked ONLY for final response rendering once a query passes the HDC similarity gate.
### What's New in v0.4 In the v0.4 release series, we tackled zero-shot extraction noise and inverted facts: - O(1) Type-Constrained Schema Validation: Set-based domain/codomain filtering over GLiREL outputs. - Automatic Direction Correction: Detects and swaps inverted extractions (e.g., turning [Budapest born_in John von Neumann] into [John von Neumann born_in Budapest]). - Precompiled Entity Sanitization: Strips possessive artifacts ('s), trailing action verbs, and prepositional tails. - Raw extraction precision improved from 11.5% to 15.5% (+35% cleaner graph).
### Current Benchmark Status (Honest Baseline) On a fixed 32-sentence benchmark (20 answerable queries, 10 hard-negative trick queries): - Retrieval Accuracy: 55.0% - Extraction Recall: 50.0% - Extraction Precision: 15.5% - Fast-Eval Retrieval Duration: ~1.27s for 30 queries (~0.04s per query) - Unanswerable Queries: 0 GPU tokens wasted (CPU similarity gate shuts down LLM calls instantly).
Repo: https://github.com/roandejager/Hillock (AGPL-3.0)
I'd love to hear feedback, criticism, or ideas from the community on hypervector binding, schema constraints, or low-power edge memory architectures!