So here is my somewhat clichéd hot take for the machine learning for creative coding class: LLMs inherently anti-creative technology. Sure, there are classification models that can do really cool things, but that’s not what I’m talking about here.

The reason LLMs are anti-creative is that they are constantly trying to say what you expect to hear. We all know at least one person who does this, and they’re rarely admired for it. These models are literally programmed to eliminate outliers. That’s why they always sound like an animated LinkedIn profile doing the worst comedy set at an open mic night.

Up until now, this has been pretty trivial stuff. But what I’m getting at is that if we eliminate outliers, we eliminate creativity, because nothing terribly creative is happening in the middle of a bell curve. That being said, wouldn’t it be cool to glitch a model so it outputs the slightly less probable? I think so.

So here is my idea: Starting small, I’d like to write a toy language model trained on a couple of thousand sentences (in the interest of performance). The input wouldn’t be a prompt—it would be a probability or confidence score. The prompt would be constant, something like “Let there be ___,” and the output would be a word generated according to the chosen probability score.

I don’t know how to do this yet, but I think it has to do with changing the learning rate and retraining the model every time. Maybe use a checkpoint before the last epoch to save time? I imagine the main challenge would be to write the code that mapped the change in learning rate into distance from the global minimum of a model. That being said, learning how to do this would be an awesome way to learn more about coding and training models in Keras.

I’m not sure what learning task this is, and the challenge would be figuring out how to use Keras differently from how it was supposed to be used in order to create something that still works.