How I build an FMV game in Unity: video, hotspots, branching

FMV gets dismissed as a dead format, and then Bandersnatch, Late Shift, Her Story, and Immortality keep proving otherwise. My angle is a director’s: the things FMV demands — coverage, performance, montage — are exactly the crafts film production already has. What Unity adds is the graph: the same footage can recombine into different experiences without reshooting.

There is also a research reason. I study interactive cinema academically, and a prototype is the only honest instrument: every directorial parameter — image density, hotspot placement, montage inserts — becomes a variable I can switch on and off inside the same scene. FMV-01 is that instrument, dressed as a noir: rain, neon, a contract, a chip on metal.

The whole slice runs on Unity VideoPlayer — one player component, sequential clips. The pattern is simple: the graph enters a Cutscene node, the node hands its clip to the player, the player pushes frames either straight to the screen or through a RenderTexture, and on the clip’s end (or a skip input) the graph moves to the next node.

// Simplified sketch of the cutscene node flow.
var player = gameObject.AddComponent<UnityEngine.Video.VideoPlayer>();
player.playOnAwake = false;
player.renderMode = UnityEngine.Video.VideoRenderMode.APIOnly;
player.url = node.clipUrl;

player.loopPointReached += (vp) => graph.Next();   // clip finished
if (skipRequested) { player.Stop(); graph.Next(); } // player input
player.Play();

Two practical notes from the slice. First, keep one player and swap sources instead of spawning a player per clip — instantiation hitches at exactly the wrong moment, between shots. Second, decide early whether frames go to the screen directly or through a RenderTexture: the texture route costs a blit but lets you keep the video inside the URP scene lighting and post stack, which matters when the video has to sit next to generated stills without looking pasted on.

The official Unity VideoPlayer documentation covers the API surface; the older Unity Discussions thread on developing an FMV adventure game is still worth reading for the failure modes, even though it predates VideoPlayer and leans on the long-dead MovieTexture.

Codec choice is usually framed as quality versus size. For an FMV game the real metric is different: how fast the player gets from “node entered” to “first frame on screen”, because in a branching story transitions happen constantly — a still look leads to a cutscene, the cutscene to another still — and every gap breaks the noir rhythm.

• H.264 (MP4) — the default choice: universal hardware decode on every platform Unity ships to, small files, fast startup. The trade-off is intra-frame seeking: landing mid-clip costs more than landing at a keyframe, so cut points should sit on keyframes.
• VP8/WebM — free codecs with solid quality, but decode runs on the CPU on several target platforms. For one foreground player it is fine; the bill arrives when several videos play or loop simultaneously.
• HAP (Hap/Qi/Hap-R) — designed exactly for the many-simultaneous-videos case: decode is moved onto the GPU, which makes heavy multi-screen rigs possible. The cost is bandwidth — huge files — and per-platform codec support. The experience report on r/Unity3D about using a lot of FMV is a useful reality check on the HAP route.

FMV-01 plays one clip at a time, so per-clip start latency dominates and a hardware-decoded, keyframe-aligned H.264 pipeline is the sane default; the codec question gets revisited only if a future scene stacks simultaneous feeds. The rule I use: pick the codec by transition latency on the weakest target device, not by compression charts.

The still frame is FMV’s secret weapon: it holds the cinematic composition of a film frame and adds interactivity without any video cost. In the slice, the look frame — the hero’s face and coat in neon rain — waits for the player. Two zones are live: the face opens a dialogue node, the coat reveals the object that feeds the hire-or-walk choice.

Technically the zones are still + UV hotspots: rectangles in the frame’s UV space mapped to graph targets, not screen-space rectangles. That distinction sounds pedantic until the frame is letterboxed, cropped, or scaled — UV zones survive all three; screen-space zones do not. The idle frame also carries a slow Ken Burns drift so the “still” never feels frozen; when the video versions of the stills are locked (the I2V step), the drift becomes a fallback rather than the default.

// UV-space hotspot test on a click.
var uv = pointerPositionInFrameUV;          // 0..1 inside the frame
foreach (var zone in node.hotspotZones)      // Still node data
    if (zone.rect.Contains(uv))
        graph.Enter(zone.targetNodeId);

Branching kills projects when every branch is a separate scene: the build doubles, then quadruples, and nothing stays consistent. FMV-01 avoids that with one graph of five node types — Dialogue, Choice, Still, Cutscene, Ending — authored as ScriptableObject data. Scenes do not own the story; the graph does. Nodes recombine, so a branch is a path through shared material, not a forked copy of the project. (The four branching structures — nodal, parallel, labyrinth, environment-based — are mapped with diagrams in Branching Narrative Structures.)

The current vertical slice is exactly this chain: intro_cs (a rain-soaked intro cutscene) → still_look (face / coat hotspots) → a hire or walk choice → ending_hired or ending_walk. It is small on purpose: the slice has to prove the loop — video, still, choice, consequence — before any content scales. The graph data being ScriptableObject also means the same file the runtime reads is the thing I can inspect in the editor, which keeps the “director brain” and the “Unity brain” in one place.

[CreateAssetMenu(menuName = "FMV/NarrativeGraph")]
public class NarrativeGraph : ScriptableObject
{
    public GraphNode[] nodes;   // Dialogue | Choice | Still | Cutscene | Ending
    public GraphNode entry;
    // Enter(nodeId) resolves the node and routes it to the right handler.
}

This is where being a director and a producer changes the Unity work. Shooting for branches is not shooting one film plus extras — coverage has to serve nodes that may play in different orders and still cut together:

• Fix the spine first. The main path (here: intro → look → choice → ending) gets full coverage; branches borrow its setups so lighting and lens stay continuous.
• Shoot every fork point as its own take with clean entry and exit — the last and first second of each variant must cut against shared frames, or the seam shows.
• Guard continuity of state. Anything the player can change (an object seen on the coat, a door opened) needs its on-screen consequence planned, or the graph must avoid referencing it later.
• Protect the cut. Transitions between clips are where FMV dies aesthetically; matching eyelines and movement vectors between variants costs nothing on set and saves the montage in the graph.
• Log takes as nodes, not as scenes. A take that can serve two nodes is a win; a scene that hard-codes one path is debt.

The longer history behind this workflow — how cut scenes evolved from external inserts into the interactive fabric — is in my research article “The Evolution of Cut Scenes”, and the platform history that shaped FMV grammar is in the 3DO FMV study.

The immediate queue, honestly: save slots (M4) so a run survives a session break, locking the I2V versions of the still frames so Ken Burns becomes the fallback, and then expanding the graph beyond the two endings. There is no public build yet — the Windows slice runs locally — and I am not going to promise a date before the save system lands. The tooling side of this process (branching editor, FMV mechanics, timeline) is a separate project: FMV Builder, which grew directly out of the hand-authoring pain described above.

If you are building your own FMV in Unity: start with one player, keyframe-aligned cuts, UV hotspots, and a graph as data — everything else is taste. Questions and corrections are welcome — the contact page is one click away, and the next devlogs will follow the save system and the still-to-video pipeline.