Author here. I ported the Doom algorithm to run inside a transformer. Instead of training a model, I used a compiler I wrote which converts computation graphs into transformer weights, and then ported Doom's algorithm into a compatible graph. The generated checkpoints can be loaded in huggingface without trust_remote_code -- it's just a standard Phi-3 checkpoint. You feed the model a prompt representing the scene data, and generate until the model stops. The result is a token sequence which includes simple pixel drawing commands (to move the cursor, draw a pixel, etc). When you mechanically apply those drawing commands you get the rendered frame.
The article includes the entire host program necessary to load the checkpoint, generate the render, and parse the output into the famous E1M1 frame. This host code is 43 lines of python. The python to define the computation graph is much longer, but that gets compiled into the transformer itself.
The post includes instructions for reproducing yourself, but unfortunately the hardware requirements are substantial. The low-res checkpoint requires 64GB of vRAM, and the 320x200 checkpoint requires a B200.
Author here. I ported the Doom algorithm to run inside a transformer. Instead of training a model, I used a compiler I wrote which converts computation graphs into transformer weights, and then ported Doom's algorithm into a compatible graph. The generated checkpoints can be loaded in huggingface without trust_remote_code -- it's just a standard Phi-3 checkpoint. You feed the model a prompt representing the scene data, and generate until the model stops. The result is a token sequence which includes simple pixel drawing commands (to move the cursor, draw a pixel, etc). When you mechanically apply those drawing commands you get the rendered frame.
The article includes the entire host program necessary to load the checkpoint, generate the render, and parse the output into the famous E1M1 frame. This host code is 43 lines of python. The python to define the computation graph is much longer, but that gets compiled into the transformer itself.
The post includes instructions for reproducing yourself, but unfortunately the hardware requirements are substantial. The low-res checkpoint requires 64GB of vRAM, and the 320x200 checkpoint requires a B200.
Happy to answer any questions!