sup computer: a small language model studio

After writing my first small language model, I found myself catching the bug. In my practice I often fixate on a new technique/process and it becomes the fuel for a burst of creative energy. This is how my pen plotter drawings started and explains the transition then to risograph prints and last year the laser cutter paper reliefs. I used to feel self-conscious about it but then I realized it is healthy for my imagination. Being able to follow my interests and spend time exploring new ways to express myself with machines.

My newest fixation is called: sup computer, a small language model studio. The focus on small models means that I can learn by doing. Instead of AI being a monolith that is trained in massive data centers on all of human knowledge, it becomes legible and steerable. I can feed the models different data sets and watch as their shape and capabilities change. Daydream was fed chess games and now it roughly knows the shape of one so you can play against it. Kenosha Kid was fed variations of a single sentence in order to prod at the usefulness of hallucinated outputs.

Each of these started with a simple question or a variation on something existing. For me this is how research starts, a fun wondering that grows and rolls up into something larger. It creates a flywheel where new knowledge unearths new paths to follow which in turn unearth new knowledge and different questions.

And the questions are the point. Are we in a bubble or not? Are we building god or a stochastic parrot? Are we racing towards our own end? I am unable to answer these now and might not ever be. But I can at least start with smaller questions. I can use these models as lights along a path towards a greater understanding and leave a trail behind me for others to follow. Because sup computer is open-source, anyone can pick it up and use frontier models to train their own models. My hope is that this helps a thousand smaller models bloom and that together we can stitch them into something larger and more meaningful than the frontier. But for now, I’m going to keep tinkering.

-mello