Chinchilla says my model shouldn't exist
Every AI team I know ships models the same way: the new one beats the old one on the eval suite, so it ships. The number went up. What else is there to know? Here's what else. For three weeks, the AI songwriter running behind my songbook was beating every metric I threw at it — and it turned out to be a cover band.
The only test that counts
I spent a week training songwriting models on the RTX 3090 in my kitchen — a 126-million-parameter GPT, built from scratch, fed 19,244 chord charts scraped out of 24,389 songs.
By the book, this model is illegal. Chinchilla-optimal — the scaling law everyone cites — says ~20 tokens of data per parameter; my 96 million tokens justify a model of about 5 million parameters. I built one twenty-five times bigger, on purpose, betting that repeated data plus ruthless early stopping would beat right-sizing. The law didn't disappear because I ignored it: in three separate training runs, learning hit the wall at almost exactly four epochs — the number the research predicts — like it had an appointment. Scaling laws are real. They're just not the same thing as a good song.
Which is why the project's rule zero, written down before anything else: no number gets a vote. Every model ships only if it wins a blind A/B, judged by me, labels hidden. Metrics are diagnostics. Ears are the verdict. That rule got more expensive to keep every day, and it was right every time.
The champion was a cover band
My reigning model — the one texting songs to real people — sailed through every eval. Then in a blind test, reading pairs with no idea which model wrote what, I flagged a lyric sheet as almost verbatim Beatles. Then another. Then Dylan's "It Ain't Me Babe", then Paul Simon. Every one of them came from the champion. It hadn't learned to write; it had learned to memorize, and my benchmarks were grading its taste in covers. It was disqualified that night, and every model since has faced a regurgitation gate: share twelve consecutive words with the training data and you're out, no matter how good you sound.
The model that won everything and lost
The next model was better by literally every number I could compute — more lyrical, better harmony, the cleanest originality profile of the project. I was ready to crown it. In the blind test I preferred the old model fourteen to five. The autopsy found half the story in ten minutes: the new training data had taught it to open songs with scraped-webpage junk, and my eyes went where the artifacts were. The other half was worse: even on clean pairs, it just wrote worse songs. Fourteen to five. No metric saw either problem coming.
The run I was told was dead
By the final model I'd rebuilt the training data from scratch — thirty-five thousand lines of junk stripped, every fix the failures had taught. Late in training, the loss curves flatlined and the forecast said the run was done learning. I said let it run anyway. Five hours later, in the last glide of the schedule, it quietly found the best checkpoint of the entire project. The machines kept the books all week; the two best calls were still human. That model — v8 — is the one writing songs today, behind a guard that checks every single output for plagiarism before it leaves the house.
So which is it?
The full build log — every failed model, every number, every lesson — is now a page of its own, and at the bottom the final model is live. Type a title and it writes you a song, if you know the password.
The whole story, and the demo, is here.
Chinchilla said the model shouldn't exist. The benchmarks said the wrong one was best. The only opinion that survived the week was the one that listened.