I built a GPT from scratch and taught it to write songs

There's a GPU in my kitchen that writes songs. Not brilliantly, but honestly, and I built every layer of it: the tokenizer, the attention blocks, the training loop, the evaluation harness and the data pipeline. This is how it went from a lookup table to a 126-million-parameter model, what broke, and why the best-measuring model I trained lost to my ears.

Building it

One concept per file: a bigram table, a single attention head, a transformer block, a full GPT, then byte-pair encoding written from scratch, trained on chord charts so it writes the sheet a guitarist wants.

The flagship has 126M parameters, 12 layers, a 1024-token context and a chord-aware tokenizer where every chord is one token. It trains on the RTX 3090 in my kitchen in about four hours.

Lessons learned

A better number is not a better model: judge by blind A/B on samples, never by loss. A blind test rewards theft, so memorized lines need detecting. And data beats cleverness, again and again.

Try it

The live model runs on the kitchen GPU behind a queue, and every sample is checked against the training corpus so it can't hand you someone else's chorus. It is password-gated because it's my GPU.