I built Picodevil using an LLM

Sign in<br>Subscribe

So this is a post about how I used an LLM to make my latest project, and how I feel about that.

To say upfront: I expect a bunch of people to be mad at me for this. Sorry! You're allowed to be mad - for the record, I do think LLMs are an unethical technology. I am not going to try to convince you that they are good. I am using them anyway despite thinking this. And I am planning on continuing to use them.

Let's get into it. Why am I doing this?

Being a creative technologist

One of the many things I call myself when I am working is a "creative technologist". To me, this means looking at new technology and going "what are interesting things you could do with this?". And — I honestly have a pretty ambivalent feeling about the job! I don't like how it involves having your agenda set by large companies with something to promote - whether you are making something that makes the technology seem cool or making something that makes the technology seem ominous, either way they are the ones setting the agenda. I don't like that feeling of rushing to claim a territory. I don't like that it fundamentally centers the technology rather than the experience or the people.

Nevertheless, it's a fun job, and I'm good at it, and people pay me money for it. I think it's normal to have reservations about your chosen field of work. I certainly have reservations about the other fields I also work in! It can be a little bit evil, but, like, really not that evil as evil jobs go.

Anyway, it turns out that the exciting new technology these days is "LLMs". And this has been actually quite convenient for me, because at university I studied... well, not large language models, they hadn't been invented back then, but certainly neural networks and autonomous robotics, and a lot of this kind of squishy non-deterministic non-symbolic AI approach. And then as an artist I have stayed connected to people working in this space - like, I followed Allison Parrish's work closely, and I played with char-rnn when that was new, and I understand what it means when you take the "vector-space embedding" of a phrase. Heck, even before university, in 2004 or so - I had an admissions interview where I got a cushy offer ("it's going to be low, but take that as a sign that we want you rather than that we're crap") partly on the basis of having spent a bunch of time messing around with chatbots.

I'm not saying all this to justify myself, really. It's more to explain where I'm coming from. It's a weird feeling. It's like I was into this cool underground band and I saw them a bunch when they were playing basement shows and then a decade later they're really commercial and are doing stadium tours. Except in this case "basement shows" is a metaphor for "well funded university research labs" and "doing stadium tours" is a metaphor for, among other things, significantly thwarting attempts to resolve the climate crisis before it makes the planet much less habitable by the human race.

Anyway, I found myself working for a startup recently, doing game design consulting for them. Usually this involves getting my hands dirty building prototypes, but in this case the founder I was working for was actually writing most of the code. Vibecoding it, actually. And that made me feel like I was fucking up at my job of "knowing the design implications of new technology". How can I confidently give advice on a piece of technology that I haven't used to someone who has? I hadn't written any major software using an LLM. And it seems extremely likely that any future startup I would work with would put me in the same position.

So, I decided to get set up, and try using an LLM to write some software. And there was this idea I'd had, that I had been kicking around for a few years, but always seemed kind of daunting. So I decided — let's try this, what have I got to lose?

The project is called Picodevil

Picodevil is a livecoding tool for making visuals from videoclips. It runs in the browser. You write... well, technically it's Javascript, but mostly you're just chaining methods together to configure how the videoclips should play. Often you're setting these parameters using a pattern language that is taken wholesale from Strudel. The whole thing is really an extension and adaptation of Strudel from the realm of audio to video.

Let's look at a little example:

$: s("ducks, canalboat").index().rows(3).cols(3).gridMod()

produces this, on a loop:

Anyway, I could go on, but this post is about using AI, not about Picodevil itself. But if you do want more detail on how it works, this readme is the best place to look. And you can also just... go off and try it out

In fact let's link that so there's a card you can click on:

picodevil<br>picodevil is here to help you livecode visuals! let’s chop up some videos with patterns!

Go try it out! I think it's more interesting than this post about how it was made.

There's a big long list...