From a Phone Video to Working Game Code

TL;DR

I developed a tool and a workflow to turn a video of gameplay, both 2D and 3D, into actual game code, using AI and a game engine. My workflow managed to create the game code of Smash Fest, a casual 3D mobile game, from about 8 minutes (total) of video, in about 1 night + early morning. This includes the level design of the levels and all the meta game screens that appeared in the videos. I did not try to recreate the art and textures, just the code, and the result was a working game.

Of course, feel free to ping me if you have any questions after reading the post.

Smash Fest gameplay

My Generated gameplay example

Background

Before I start, I should mention that I use a personal game-development harness I call PlayDreams. It lets me and a few friends develop games faster and more reliably across Unity, Godot and web, and also covers art creation and deployment to the mobile stores. Some of the processes I describe below use tools and systems I already have there, but the video-to-game process itself that I describe here should be fairly clear, and a developer using Claude or Codex should be able to reproduce the same general workflow in their own setup.

I recently added a tool to turn a video of a character into a 2D sprite to PlayDreams, my game development harness. The tool was originally developed by a friend, and as I played with it I started to wonder if I could turn it into a full video-to-game-code workflow. So I gave it a shot.

The game conversion I’ll detail below is turning a video of Smash Fest into the actual gameplay (and a few levels), but to be fair, I first developed the system on a 2D game - my own 2D Mahjong game.

This is important since I trained the system first on a game I own the code for - the orchestrator had access to the actual source code and could direct the first attempt to turn a video into game code fairly independently, and correct its own mistakes independently. I believe this initial process was critical to getting the base code right, especially since I wanted the process to be as code-based (vs LLM-based) as possible. Even though I used frontier models (Claude’s Opus and Fable with reviews of OpenAI’s Terra and Sol), they still made quite a lot of mistakes (but with confidence!) that only the comparison with the actual code floated.

This created a good foundation, even though the system was calibrated to 2D games and assumed no depth (which made analysis much easier). In this post I’ll describe the process I used when developing and implementing it on a 3D game - but an easy one. Smash Fest has fixed-camera gameplay, a single screen (no scrolling), and is physics-based (something fairly easy for most game engines).

Why I did it in Godot

I chose to do the process with Godot. The development cycles are MUCH faster than Unity, the engine is more than capable and stable enough to handle everything needed for the game and has structural scenes and assets. Also, my PlayDreams harness has a very strong Godot development setup, which made development easier and faster.

Regardless, the process I describe is agnostic to how you implement the game code.

I’ll walk through it in order, and at the end I’ll say which parts are done by plain code and which parts actually need an LLM, because that distinction turned out to matter more than I originally expected.

Step 1 - Figure out what to film.

For proper analysis I needed a good video (or more than one in later stages). It needed to cover everything I wanted to develop - that means all the menus, all the game states - level fail, level win, use of pre-level booster, what happens if I’m out of hearts etc. For every dimension of the game (camera, physics, level structure, progression, UI).

Step 2 - Write down what the video can’t tell you

This is an important step.

Before any code, I produce an absence list: a screen-by-screen inventory where every gap is classified as one of four things.

Resolution - it’s on screen but too small or too blurry to measure.
Coverage - it exists in the game but never appears in the footage.
Unfilmed - it exists and you simply didn’t record it.
Observability - it can never be seen from outside. Drop rates. Difficulty curves. Anything server-side.

Coverage gaps hinder production and send you back to recording. Observability gaps can never be closed by more footage, so you stop pretending and go to research instead.

For example, the video never indicates where the player’s finger is. In the game, the cannon aims and shoots where the finger touches the target, but without that, it is not visible in the video. So some things just need to be said first - or instruct the model to research the description/design of that game on the net (and review it since every little wrong assumption can turn into a significant design problem in implementation).

Another example specific to Smash Fest - the levels are built from layers (rows) of 3D objects stacked on top of each other, one behind the other. You can never see the back layers, unless you find a way to make the front layer fall alone or the whole thing is on a rotating table (some levels are like that). This is an issue to handle.

The reason this matters: without it, the LLM model will confidently invent the parts it couldn’t see. Sometimes it will be right, and sometimes - very wrong.

Step 3 - Turn video into evidence

Now I extract frames.

For the video processing I use ffmpeg. I use it for the mechanical work around the footage - extracting frames, sampling different parts at different rates, and keeping everything tied to the original timestamps. The LLM is not doing that work - it gets the resulting frames and evidence to analyze.

But you do not need every frame at the same density. Different things need wildly different sampling:

A menu or a screen layout - One frame. It’s static. A second frame gives no additional data.
Level structure and layout - About one frame every half second. Enough to see it change, not so much you drown in details (and waste tokens and time).
VFX, impacts, camera shake, feel - Full frame rate. Especially if you have effects that include fast particles, shooting, colors cycling etc.

The segmentation tools split the footage into named spans (a stretch of footage gets an id, so you can cite a window instead of re-deriving it), classify camera motion per span (there’s deliberately no single global camera model - a menu and a gameplay moment don’t share one), and separate screens from gameplay from effects.

Every number I take off the footage gets a measurement record - an id, a frame citation with clip and timestamp, the resolution it was measured at, and a round-trip back to pixels. So a constant sitting in the code can be traced to the exact frame that produced it. Uncited number means invented number (which is something you may need to have in some places, but our goal is to minimize these as much as possible).

One important note - this stage can consume A LOT of disk space. I used video captured from my iPhone, and it was high-res with 60fps. Several minutes of it. I ended up with over 20gb of frames on my hard drive. But that is the cost of analyzing videos (and letting the agents run loops independently).

Level Analysis - Special issue

Another important potential failure point - level analysis.

The LLMs have a tendency to look at the initial frame of the level and deduce that this is the level data. This is almost always WRONG. In most games a level is a temporal object, not a spatial one. You cannot look at the first frame of a level and know what the level consists of. The starting frame doesn’t tell you what falls, what spawns, what the win condition is, or how it ends. In almost every game genre game elements appear as the level progresses - screen scrolling reveals more enemies, more platforms, more power ups, level architecture, etc. In match 3 games the board changes with chain reactions and elements added. In Mah Jong tile faces are revealed only when the tiles above them are taken. You must observe the whole level played (sometimes more than once) to fully get what’s in there.

As I mentioned, I chose “easy” games to train my system. Both in Mah Jong and in Smash Fest, even if some elements are occluded at the beginning - they are all there from the start of the level.

Analyzing several levels is also important so I could understand what are the building blocks, what is fixed, and what are parameters that define a level. This is super important if I want to generate new levels going forward.

Step 4 - The second input channel: genre research

Footage is one channel. Research is the other, and it’s an important side.

Whether you want it or not, your LLM is going to look up things in its own knowledge base and search the internet to interpret things it finds. That’s how LLMs work. You could disregard it and hope it will do good research on its own (which in many cases it will) or you can formalize the process so you’ll have some control over it.

In parallel to a subagent extracting the frames from the video, another subagent is tasked with writing a design brief document. It starts with looking up the game name or getting a general game description, and then looking up the genre in my game-design knowledge base and on the web: what is the standard core gameplay for this kind of game, how is a level normally played, win/lose conditions, controls, what the conventional progression looks like. Every claim from this channel is tagged research and carries a citation. An uncited research claim counts as an invention, exactly like an uncited measurement.

This is not a GDD (game design document) - this is a shorter list of points describing elements of the game design. No flow charts, no progression loops, nothing that organized. This is all generated.

The split ends up being: the footage tells you what this game does, research tells you what games like it normally do, and where the footage is silent, research fills in - visibly, with a tag on it.

In Smash Fest’s case, for example, I didn’t have the subgenre in my knowledge base so everything had to come from the video and web search.

Step 5 - Analyze the game genre and write docs

The system is designed to write basically 2 documents: the game design brief, and the technical implementation plan created once the design brief is approved. Both must be done in sequence (since technical design is based on the design brief) and both are approved by a human (me) before implementation starts.

Since I wanted the development to run in a loop until completion without me being in the loop - this part was critical to get right. I read the design brief (I’ve been a VP Product in game studios most of the past decade, reviewing designs is my comfort zone…) and made corrections, and after the technical plan was ready I read, corrected and approved it as well. This was probably the part I had the most work to do in this whole video to game workflow.

Step 6 - Art Guidelines: deliberately do less

My basic guideline for this experiment was don’t put effort into extracting the art.

The reference art is copyrighted. Every asset that comes off those frames is a placeholder that gets thrown away before anything can ship. Polishing it is pure waste of time and tokens.

So the rule is: grab whatever is easy to extract, and put your effort into building the game easy to swap assets in and out of. Asset for asset. The structure was more important than what the actual texture or image actually was (since you can use NOTHING of the original art).

Every visual element is a slot with a declared size and role. A slot starts as a colored box at the correct dimensions. There’s a validator that checks whether a replacement asset actually conforms to its slot - and identical canvas dimensions are not enough to pass, because a correctly-sized image with the content in the wrong place still breaks the layout.

Structural replaceability is the actual art deliverable of the first pass. The pretty version comes later, from a real artist or a genAI model, and it should be able to be dropped in without touching a scene.

Personally, I also have all of Kenney’s CC0 packs (which I happily paid for) in a personal MCP which I use for placeholder art. I also took the time a long time ago to categorize and describe each element and package so the asset bank is easily searchable.

So I have a cost ladder for filling slots - extract, procedural, asset bank, generated - and every filled slot has to name which rung filled it. For this case I forbade generating art - it would extract, make procedural art or find a matching CC0 asset in the asset bank.

Step 7 - Implementation using a compare loop

This is the point where the system starts working independently. We have video footage, we have approved complementary documents and art guidelines.

Now we start the core development loop:

develop or fix code -> runtime gate (does the game boot with zero errors) -> capture a screenshot of a deliberately constructed game state -> compare -> classify defects -> repeat.

I provided a clear goal to the orchestrator agent - I want to see a functioning, playable game (minus exact art) where each level plays exactly like in the video. So it needs to loop over and over until the level is identical, the elements behave the same and with the right timing. So up to half a second up or down in timing is fine, but otherwise it means something is off. The implementation, a parameter, whatever. Continue the loop until it behaves the same. Looping development without proper goals will get you nowhere. It will either stop too early and claim done, or it will continue past done and will mess up the code (or burn through every token you have).

Two different comparisons run, and both are needed because they catch different things:

Reference comparison grades my capture against pinned footage frames on declared, measurable invariants - layout, geometry, gross palette, whether the right icon is bound to the right role.
Per-asset audit grades the assets themselves for content.

This split isn’t a guess - it was measured on the earlier 2D mah jong game I ran through the same pipeline, where the structural comparison passed nearly every contaminated asset it was shown. Structure-level checking is blind to asset-level rot: a screen can have every box in the right place and still be built out of the wrong pixels. Run both.

Two constraints on the loop that matter:

It’s bounded. Three fix attempts per feature, then it escalates to a stronger, more expensive model, up to 2 attempts there and if it continues to fail - it needs to escalate to a person. No infinite grinding.
The order is fixed so no capture ever grades the filter that produced it. If you build a check and then capture through it, you’ve proven nothing.

Step 8 - Gates

Two extra gates I added on this route.

The game must be tested from a player’s perspective. Launch the real scene, send no input, and fail if the picture permanently and broadly changes on its own. Permanence is the test, not motion - idle animation and particle loops pass fine. This exists because pressing Play once gave me a test harness driving itself instead of a game. Pressing Play has to hand the player the game.

No asset slot may hold greybox. Every image slot names the rung that filled it, and each claim is checked against the file’s actual pixels, so a grey box can’t wear a provenance stamp. It also reads the project’s scenes to check the art is actually reachable - a slot no scene references is an orphan, and a slot whose scene loads a different flat file is caught separately.

This also came from issues I had when converting the mah-jong video into game code - it once passed every single slot on a build that had never been compared to a reference frame at all. A clean asset manifest is not a visible game. It only tells you a real file reaches each slot through a path the engine loads. It is not resemblance and it is not art review.

Step 9 - The GDD comes last

This sounds backwards but this is how I chose to do it.

The GDD is written at the end of the process, reconciled against the game that actually exists. Every mechanics claim in it carries where it came from: [observed] from the footage, [inferred], [user-specified], or [unknown-unfilmed]. A checker refuses the document if any claim is bare.

So you end up with a design doc where you can see, per line, whether it’s a fact or a guess. I think this is critical this way because - let’s face it, it will never be exactly what you want in the first version. You will have to fix things. Having a GDD that is TRUE to what you have as a starting point is super helpful since after so much independent work, you have a document that you can review to know REALLY what is implemented, and where to begin changing.

Who does what: code vs LLM

This is the part I’d most want a technical reader to take away. Most of this pipeline is not an LLM. Most of it is boring deterministic code, and that’s on purpose - anything with a decidable answer is a script, a schema, or a test, never a sentence asking a model to remember something.

Plain code, zero LLM:

Frame extraction, frame-rate normalisation, hashing and pinning clips
Span detection, scene-cut detection, per-span camera-motion classification
Screen segmentation and the unoccluded-face sweeps
The measurement record - IDs, citations, coordinate round-trips
Absence-list validation and the coverage arguments
Slot definitions, greybox emission, slot conformance validation
Screenshot capture and the boot smoke test
The reference comparison and the per-asset audits
The player-driven and art-wired gates
The leakage canary
Provenance checking on the design brief
The route verifier - one manifest, one run, one result, and there is deliberately no way to type a verdict by hand

Needs an LLM:

Looking at frames and saying what’s on screen - and it reports back in text only, with confidence and a frame citation per claim, never by pasting images upstream
Deciding what the game’s rules probably are from what was observed
Genre research and pulling the conventions together
Writing the design brief, the art direction, the game design document at the end
Writing the actual (Godot) code
Judging whether a visual difference matters or not
Deciding which of the flagged defects to fix first

The pattern: the LLM reads, judges and writes. The code measures, refuses and records. Whenever I caught the model getting something wrong repeatedly, the fix was never to word the instruction more firmly - it was to turn the rule into a script that fails.

My Takeaways

Don’t treat frames as one pile. Screens need one frame. Level structure needs roughly one frame per half second (depending on game genre). VFX needs full frame rate. Sampling all three the same way means you either drown in data or miss the effects entirely.
A level is a temporal, dynamic entity. LLMs sometimes tend to deduce a level design from an initial frame. Levels in most games include many elements that are only revealed as you play. You need to analyze a video of a level from start to finish to get the data needed to build it.
Don’t over-work the art. It’s copyrighted, it’s temporary, it’s getting deleted. Grab what’s cheap to extract and move on.
Spend that effort on replaceability instead. Slots with declared sizes and roles, and a validator that checks a replacement actually fits. Asset-for-asset swapping is the deliverable.
Say what you can’t know, before you build. The absence list is the highest-value hour in the whole process. Gaps you name become filming tasks; gaps you don’t name become inventions.
Two comparisons, not one. Structure-level checks wave contaminated assets straight through. Every box in the right place, wrong pixels inside them.
Order your checks so nothing grades itself. Capture before the filter exists.
A passing gate is not a resemblance claim. Write down what each check doesn’t say, in the check’s own output. Otherwise “every slot passed” may be understood by the LLM as “the game looks right.”
Bound your loops. Three attempts, then escalate. To a better model (which you bound as well) or directly to a human. An unbounded improve-until-good loop just burns money with little contribution.
Write the full design doc last, with provenance on every line. So you have a solid starting point to start iterating.

The whole thing as one picture

Boxes in blue are plain deterministic code. Boxes in amber are where an LLM reads, judges or writes. Diamonds are the points where the process stops and asks a human.

flowchart TD
    A["Reference game<br/>name, developer, store links, phone video"]:::human --> B["Step 1 - Capture brief<br/>what to film and what each clip should reveal"]:::code

    B --> C["Step 2 - ABSENCE LIST<br/>coverage / resolution / unfilmed / observability"]:::code

    C -->|coverage or unfilmed gap| D["Film more"]:::human
    D --> B

    C -->|covered enough to proceed| E["Step 3 - Video processing<br/>extract, sample, timestamp, segment"]:::code

    E --> E1["Screens<br/>~1 frame each"]:::code
    E --> E2["Level structure<br/>~1 frame / 0.5 s"]:::code
    E --> E3["VFX / impacts / feel<br/>full frame rate"]:::code

    E1 --> F["Build evidence<br/>spans, camera regime, temporal level data,<br/>measurements with frame citations"]:::code
    E2 --> F
    E3 --> F

    F --> G["Read and interpret frames<br/>what is visible, with confidence + citation"]:::llm

    C -->|observability gap| H["Step 4 - Genre / game research<br/>knowledge base + web<br/>tag claims as research"]:::llm
    G --> H

    G --> I["Combine observed evidence<br/>with research / inference"]:::llm
    H --> I

    I --> J["Step 5 - Write design brief"]:::llm

    J --> K{"Human approves<br/>design brief?"}:::gate
    K -->|no| J
    K -->|yes| L["Write technical implementation plan"]:::llm

    L --> M{"Human approves<br/>technical plan?"}:::gate
    M -->|no| L

    M -->|yes| N["Step 6 - Define art guidelines + SLOTS<br/>declared size and role<br/>extract / procedural / CC0 / generated"]:::code

    N --> O["Step 7 - Build initial game implementation"]:::llm

    O --> P["Boot smoke test<br/>zero runtime errors"]:::code
    P --> Q["Capture constructed game state"]:::code

    Q --> R1["Reference comparison<br/>layout, geometry, palette, role binding"]:::code
    Q --> R2["Per-asset audit<br/>asset content"]:::code

    R1 --> S{"Defects?"}:::gate
    R2 --> S

    S -->|yes, attempts remain| T["Judge what matters and fix<br/>bounded retry loop"]:::llm
    T --> P

    S -->|retry budget exhausted| U["Escalate<br/>stronger model, then human"]:::gate

    S -->|clean| V["Step 8 - Final deterministic gates<br/>player-driven test<br/>no slot holds greybox<br/>art reachable from scenes"]:::code

    V --> W["Route verifier<br/>one manifest, one run, one result"]:::code

    W --> X["Step 9 - Write GDD LAST<br/>reconcile against implemented game<br/>tag every claim:<br/>observed / inferred / user-specified / unknown-unfilmed"]:::llm

    classDef code fill:#1e3a5f,stroke:#4a9eff,color:#e8f0fe
    classDef llm fill:#4a3b1a,stroke:#e0a83c,color:#fdf6e3
    classDef gate fill:#4a1f1f,stroke:#ff6b6b,color:#ffe8e8
    classDef human fill:#2a2a2a,stroke:#999,color:#eee

Two loops are worth noticing, because they’re the ones that actually cost time. The outer loop - absence list back to filming - is why one video became several. The inner loop - capture, compare, fix - is bounded at three attempts before escalation, rather than grinding forever on something it can’t see.

And notice where the amber boxes are. The LLM reads frames, decides what the rules probably are, writes the docs, writes the game, and judges what matters. Everything else - extraction, sampling, measurement, comparison, the gates - is code that either passes or refuses. That ratio is the point.