My first language model: shakespeare-nanogpt

William Shakespeare | The Poetry Foundation

ROMEO: Thou hadst to do it.

The quote above is fake Shakespeare that was written by a small language model. One that I trained at home on my personal laptop. It’s called shakespeare-nanogpt.

The idea for this project came from When AI builds itself, one of the last things I worked on at Anthropic. The core argument is: if coding agents get good enough at writing code, then they can write the code that creates another language model. This is the idea of the moment, and neolabs are spinning up with the goal of creating AI researchers. I got a glimpse of this when building sprite, my coding agent, which I wrote almost entirely with Claude Code. With shakespeare-nanogpt I wanted to push this further and let Claude Opus 4.8, a big model, train something small enough to run on an M5 Mac. I started with Andrej Karpathy’s nanogpt and built an environment where Claude could train and iterate on a smaller model. The end result is a model factory that Claude can operate to conduct experiments, write-up research findings and even generate a model card complete with data visualizations.

If you believe that the future is less a giant all-knowing god model and more a garden of smaller models orchestrated and working together, then this path is exciting. As an artist, this is an extension of my “mixing my own paints” philosophy. An attempt to ensure that I own and understand each part of my technology stack. It’s funny that the moment I leave Anthropic, I become more interested in how the underlying technology works, but there’s never been a better time to learn. My hope is to turn this research into novel interface experiments that I can showcase here but for now, we start with fake Shakespeare.

-Mello

P.S. - All praise to Apple and the M series chips. The fact that I can train a language model while still writing code, browsing the web and running several different applications feels futuristic.

P.P.S - Full output from the quote above. 250 tokens

ROMEO: Thou hadst to do it.

Nurse: Now, it did, my good is too much less.

**JULIET: **This day’st thou not married thee this land,

Not to thine eyes is little and day dead,

The courtesy liberal that love’s sweet day’st thou?

The morning of thy bold and thee to the prince’?