I built a small language model in Minecraft using no command blocks or datapacks!
The model has 5,087,280 parameters, trained in Python on the TinyChat dataset of basic English conversations. It has an embedding dimension of 240, vocabulary of 1920 tokens, and consists of 6 layers. The context window size is 64 tokens, which is enough for (very) short conversations. Most weights were quantized to 8 bits, although the embedding and LayerNorm weights are stored at 18 and 24 bits respectively. The quantized weights are linked below; they are split into hundreds of files corresponding to the separate sections of ROM in the build.
The build occupies a volume of 1020x260x1656 blocks. Due to its immense size, the Distant Horizons mod was used to capture footage of the whole build; this results in distant redstone components looking strange as they are being rendered at a lower level of detail.
It can produce a response in about 2 hours when the tick rate is increased using MCHPRS (Minecraft High Performance Redstone Server) to about 40,000x speed.
————————————————–
World download (Vanilla, 1.20.4): https://drive.google.com/file/d/1pf1PtSqCLkHPj-vGpdOxRhDcug8rLIl0/view?usp=sharing
World download (MCHPRS): https://drive.google.com/file/d/1GToQikx1VpJSOg5unnnFhJl8onq0DRhY/view?usp=sharing
Emulator, weights and guide to running the model: https://github.com/sammyuri/craftgpt
MCHPRS: https://github.com/MCHPR/MCHPRS
————————————————–
Music: TheFatRat – Fire
Watch the official music video: https://tinyurl.com/tfrfire
Listen here: https://thefatrat.ffm.to/fire
Follow TheFatRat: https://ffm.bio/thefatrat
Thank you to my sisters, debbieuri and sofiuri, who were there through thick and thin (but mostly thin).
