I’ve been working on a physics project that unexpectedly led to a new transformer architecture.
*TickBlock* replaces QKᵀ with a learnable banded positional operator (“tensor mode”).
On Tiny Shakespeare, it matches GPT-2-small-level performance with only 0.64M parameters (≈0.5% of the size). It trains in ~12 minutes on a Mac laptop — no kernel optimizations yet.
The design is motivated by my research in fundamental physics, which reinterprets relativity without tensors. That work (Project Belgrade) models reality as a sequence of discrete publishing events and photons as standing sheets of information:
[Structure and Mechanics of Reality: Project Belgrade](https://doi.org/10.5281/zenodo.17191747)
Right now the results are small-scale, but the efficiency gains suggest a bigger question:
- With kernel optimization, parameter sweeps, and compression, how far can this scale?
- Could we see GPT-4-class assistants running fully on laptops? GPT-3.5-class reasoning on phones?
- If that became practical, what would be the most compelling on-device applications?
Curious to hear how the community thinks about the tradeoffs and opportunities.
I’ve been working on a physics project that unexpectedly led to a new transformer architecture.
*TickBlock* replaces QKᵀ with a learnable banded positional operator (“tensor mode”). On Tiny Shakespeare, it matches GPT-2-small-level performance with only 0.64M parameters (≈0.5% of the size). It trains in ~12 minutes on a Mac laptop — no kernel optimizations yet.
Repo: https://github.com/projectbelgrade/tickblock
The design is motivated by my research in fundamental physics, which reinterprets relativity without tensors. That work (Project Belgrade) models reality as a sequence of discrete publishing events and photons as standing sheets of information: [Structure and Mechanics of Reality: Project Belgrade](https://doi.org/10.5281/zenodo.17191747)
Right now the results are small-scale, but the efficiency gains suggest a bigger question:
- With kernel optimization, parameter sweeps, and compression, how far can this scale? - Could we see GPT-4-class assistants running fully on laptops? GPT-3.5-class reasoning on phones? - If that became practical, what would be the most compelling on-device applications?
Curious to hear how the community thinks about the tradeoffs and opportunities.