From generative video to programmable video

When most people say “AI video” right now, they mean one thing: you type a prompt, a model thinks for a while, and a clip comes back. It’s impressive. A few sentences in, a few seconds of footage out.

But there’s another way to use AI to make video, and it looks more like web development than filmmaking. Instead of asking AI to generate the frames, you ask it to write the code that generates them.

# two ways to make a video
 
              INPUT       STEP        STEP        OUTPUT
generative    prompt  →   model ············  →   pixels
code-based    prompt  →   code    →   frames  →   video
--------------------------------------------------------
generative    ≠ run it again, get something different
code-based    = run it again, get the same frames

A generated clip is a finished artifact: if it’s not quite right, you ask again and get something different. Code renders the same frames every time, so you can keep working on it, nudging it until it’s exactly right and changing only what needs to change. That predictability is what makes this approach useful for professional work, and AI coding tools are what make it practical.

What if a video were just a website with a timeline?

Remotion has been doing this for years, letting developers build videos out of the same React components they use for websites. More recently, HeyGen open-sourced HyperFrames, which does the same thing with plain HTML and CSS. Either way, a browser draws each frame, and the frames are stitched together into a video.

Both rest on the same idea: the browser is an extremely capable graphics engine.

It already renders typography, illustration, animation, 3D, images and video. All it’s missing to become a video renderer is a clock it can step through one frame at a time. Give it one, capture what’s on screen at each step, and you have a video.

If you’ve written React, you know the mental model. You don’t reach in and change an interface; you describe what it should look like for a given state:

# UI is a function of state
UI = f(state);

Video works the same way, with time in place of state:

# frame is a function of time
frame = f(time);

You don’t tell a title to fade in. You describe where it should be at any moment: invisible at frame 0, halfway there at frame 15, fully in place by frame 30. Ask twice and you get the same answer.

Rendering, not rolling the dice

Ask a generative model for the same clip twice and you get two interpretations. Render the same code twice and you get the same frames, down to the pixel.

You can tune a coded piece the way you’d tune a layout: try something, look, adjust, look again. Every round builds on the last, and nothing you’ve already gotten right drifts while you work on something else. You stop when it’s right, not when you get lucky.

With a generative model, each revision is a new roll of the dice, and there’s a real chance the thing you liked doesn’t survive.

That matters even more once a client is involved, because revisions on a motion piece tend to be mundane:

• Hold this frame for another second.
• Ease the title in more gently.
• Replace this headline.

With code, each request is a surgical edit: a duration, a curve, a line of text. Everything the client already approved stays exactly where it was. You’re not spinning up a black box again and hoping for the best. You’re changing one thing and rendering again.

One video becomes a hundred

When a video is a function, you can give it different inputs:

<Promo title="Poetry in Motion" theme="blue" format="vertical" duration={12} />

Change the title, theme, format or data and render again.

That’s where a lot of tedious production work lives: a campaign with a dozen headlines, the same piece laid out properly at 16:9, 1:1 and 9:16, a card for every speaker on the schedule, this month’s numbers animated in the same style as last month’s. None of it is creative work, but someone usually does it by hand. Here it becomes a list of inputs.

Nobody wants to hand-edit 400 timelines. A design system for video turns out to look a lot like a design system for the web.

Why this matters now

If all of this has been possible for years, why does it matter now?

Two arcs have lined up at just the right time. Browsers have spent the last decade quietly becoming world-class graphics environments: WebGL and WebGPU put the GPU to work, WebAssembly runs code at near-native speed, and libraries like Three.js make real-time 3D possible on a web page.

At the same time, AI models have gotten remarkably good at writing the code that drives all of it. You can suddenly wield a pretty intimidating graphics stack without knowing what a quaternion is—or even really understanding how the code works.

This isn’t a replacement for generative video

This isn’t a “code good, generative bad” argument. The two are good at different things.

Code excels at things with structure: typography, UI, diagrams, charts, logos, geometric animation, product graphics, and anything that has to be exactly the same every time.

Generative video excels at what’s hard to describe procedurally: people, environments, cinematic imagery, organic movement, realistic camera work.

When the project calls for it, use it as an input to the system:

# the model supplies texture; code supplies structure + control
 
STEP   LAYER                         BY      SUPPLIES
[1/3]  generative assets / footage   model   texture
[2/3]  deterministic composition     code    structure
[3/3]  typography + graphics + data  code    control
-------------------------------------------------------
  ✓    video                                 rendered

Let a model generate the footage, then bring it into a coded composition where the titles, graphics, branding and timing are exact and editable.

The generative part supplies the texture; the code supplies the structure and control.

A small experiment

Arguments are cheap, so I tried it: the stack above in miniature, with AI-generated footage under coded layouts. I gave a coding agent a brief and a folder of assets and had it build the piece as a plain web page, with no video framework at all. A browser stepped through the frames and they were encoded into a video, which took about 90 seconds for a 6-second render.

The brief: three event promos (a disco night, a data science talk and a game jam), each already designed as a 16:9 cover and a 4:5 Instagram post. I wanted one seamless loop of about 6 seconds that cycles through all six. Each graphic floats as a card over a blurred copy of its own background video, and at every change each element moves and scales to its place in the next layout. The inputs were:

• the six layouts, as PNG exports and as HTML source from the design tool
• three AI-generated 5-second background loops
• one screenshot of the floating-card look

Here’s how it went:

1. Plan it: “Here are the assets. Build this loop, plus an admin view to review it.” It initially offered to rebuild the layouts from the PNGs, but I pointed it at the original design source instead. It ported all six with their styles intact, and each matched its PNG export on the first render.
2. Make it move: The first render held every layout perfectly and fell apart in between. “Disco Night” sits on one line in 16:9 and two in 4:5, so the title cross-dissolved into a smeared double image. Text also ran ahead of the card as it resized and got clipped at its edge. The fix was structural: each title became individual words that fly to their new positions, and every element now moves with the card.
3. Swap the footage: When the final background renders arrived, swapping them in meant changing one file path per event and rendering again.
4. Adjust the timing: I wanted each layout held longer and the transitions slower, so I did this one myself in the admin view: hold from 0.4 to 1 second, transition from 0.6 to 2, then render. No code changed, just two numbers in a settings file.
5. Generate variations: These were already built in: one set of templates, three sets of event details, two aspect ratios. A fourth event would be one more entry in a list.

The layouts worked on the first try because the agent had real design source instead of screenshots. The motion took two rounds of fixes. The agent found those problems by rendering and checking still frames from the middle of each transition. The fixes changed the motion without disturbing the layouts that were already right.

Control is the point

Professional video work is a process. There’s a brief, a first cut, feedback, a second cut, a client who loves it except for the logo, and a dozen different formats due Friday. That process depends on getting something right and keeping it right.

Until now, AI video has mostly been a parlor trick: impressive on the first take, hopeless by the second round of notes. Build the video as a system and you can use AI for everything it’s good at without giving up any control. You’re art directing, not rolling the dice.