FrameWorkers — Research

FrameWorkers is a multi-agent system

Instead of one model doing everything, video production is split among specialized AI agents, each owning a single job. A trained Director plans the production and re-plans on failure, an Assistant executes every step in a shared workspace, and a crew of specialist sub-agents does the creative work — replacing a fixed pipeline with a closed, recoverable loop.

Why not a fixed pipeline

Real video-making spans scripting, asset management, and cross-shot consistency — and different inputs need different workflows. A hand-wired pipeline is brittle; a zero-shot LLM struggles with long-horizon planning and multimodal asset routing. FrameWorkers instead treats video creation as dynamic task management: every task flows through one closed loop.

1

Plan

The Director edits the task stack from the intent, agent descriptors, and current state.

2

Execute

The Assistant resolves inputs from the workspace and invokes the right specialist agent.

3

Check

Outputs are parsed, validated, and written back as assets, memory, and logs.

4

Re-plan

On failure, the Director revises the remaining tasks — recover, don't restart.

Director orchestrating a dynamic task stack in a self-correcting loop

Module 1 · Planning

The Director & the dynamic task stack

The orchestrator. It reads the brief, the agent descriptors, and current state, then edits a dynamic task stack — adding, reordering, and re-planning on failure, never a hard-coded graph. And it is trained, not just prompted: fine-tuned with SFT + RL so it picks the right agents, orders them by dependency, and recovers from failures.

Director routing accuracy

Given a request, does the Director pick the right specialist agents and put them in the right order? We compare its plan against a known-correct agent chain on held-out test cases and count only an exact match. Ours is a Qwen3-8B model fine-tuned with SFT + GRPO.

FrameWorkers (ours)

88.4%

Gemini 3 Pro Preview

71.2%

Gemini 2.5 Pro

66.3%

Qwen3.6 Plus

55.9%

GPT-4o

50.5%

DeepSeek-V3

39.4%

Qwen3-8B (base)

24.5%

Qwen3-4B (base)

12.5%

Runtime recovery

If a step fails partway through a production, can the Director recover — build a new valid plan from where things broke — instead of the whole run failing?

99.8%

Re-plan · missing step

A required producer task is removed; the Director detects the broken dependency and inserts the missing step (plan-level, 495 cases).

93.5%

Replace · rejected output

A quality gate rejects an artifact; the Director selects a corrective step — regenerate or refine — before continuing (execution-level, 496 cases).

96.7% overall recovery across 991 perturbed executions — with no explicit recovery training.

The Assistant routing inputs and outputs through a shared workspace

Module 2 · Execution

The Assistant & the workspace

The execution runtime. It resolves each task’s inputs, invokes the chosen agent, validates the output, and persists everything to a shared workspace — assets, global memory, and logs. A “strict-out, loose-in” contract keeps agents decoupled and recoverable.

Execution robustness

Once the Director has a plan, can the Assistant actually carry it out end to end — every agent call completing, every output validating, nothing lost to a workspace error? Case-level pass rate on a 204-case stratified suite, by the LLM backend powering execution.

Gemini 2.5 Flash

100%

Gemini 2.0 Flash

91.7%

GPT-4o-mini

81.4%

GPT-4o

61.8%

Gemini 2.5 Flash — the backend FrameWorkers ships with — completes every case with zero orchestration, routing, or workspace-level failures.

A pluggable catalog of sub-agents with a subset selected into a chain

Module 3 · The crew

A catalog of sub-agents

Capability as a catalog, not a graph. Each sub-agent is a self-contained module owning one capability, described by its inputs, outputs, and triggers. Routing is semantic, so new capabilities are added by registering a descriptor — no workflow redesign. The Director selects and orders a subset per production; the agents below are in active use today.

Narrative Travelogue Explainer News Broadcast Advertisement Adaptation Comedy Keyframe Sheet Shot Prompt Clip Narration Storybook Illustration Narrator Voiceover Music Ambience Audio Mix Video Analysis Highlight Style Transfer Video Extend Upscale Brief Enricher Image Intake Video Intake Transcription Translation Compositor

Capability coverage

Can the system take this kind of input, and can it produce this kind of output? This checklist compares what each system explicitly supports — not how well it does it — against six published multi-agent systems. A cross means not explicitly demonstrated in that system's paper, not necessarily an inherent inability.

Property	FrameWorkers (ours)	UniVA	Co-Director	MovieAgent	Anim- Director	AniMaker	MM-Story Agent
Supported inputs (each row is in addition to a base instruction)
Instruction only	✓	✓	✓	✗	✓	✓	✓
+ raw source text	✓	✓	✗	✗	✗	✗	✗
+ script	✓	✗	✗	✓	✗	✗	✗
+ reference image(s)	✓	✓	✓	✓	✗	✗	✗
+ video asset(s)	✓	✓	✗	✗	✗	✗	✗
+ audio asset(s)	✓	✓	✗	✓	✗	✗	✗
+ script + reference image(s)	✓	✗	✗	✓	✗	✗	✗
+ script + audio asset(s)	✓	✗	✗	✓	✗	✗	✗
+ reference image(s) + video asset(s)	✓	✓	✗	✗	✗	✗	✗
+ script + reference image(s) + audio asset(s)	✓	✗	✗	✓	✗	✗	✗
Output properties
Multi-shot dynamic video	✓	✓	✓	✓	✓	✓	✗
Image-sequence / storybook video	✓	✗	✗	✗	✗	✗	✓
Refined existing video	✓	✓	✗	✗	✗	✗	✗
Speech / voice-over audio	✓	✓	✓	✓	✓	✓	✓
Music / Foley / sound effects	✓	✓	✓	✗	✗	✗	✓
Subtitles	✓	✓	✗	✓	✗	✓	✗
Final audiovisual composition	✓	✓	✓	✓	✓	✓	✓
Total	17/17	12/17	6/17	10/17	4/17	5/17	5/17

Add a new specialist — no retraining

A fixed pipeline must be redesigned to gain a new capability. Here we register two specialists the Director never saw during training — by dropping in a descriptor only — and measure whether it selects them correctly and whether the resulting production runs.

ComedyAgent

unseen · added post-training

93.0%

correctly selected

99.5%

workflow passes

ScreenwriterAgent

unseen · added post-training

99.2%

correctly selected

88.8%

workflow passes

The Director generalizes to new sub-agents from their descriptor alone — capability grows by registration, not redesign.

Full system

End-to-end results

Do the three modules add up to better films? The finished videos themselves, scored blind against four external baselines (S-Agent, UniVA, Anim-Director, MovieAgent) across two studies — by a multimodal LLM judge and by human evaluators.

End-to-end video quality — automatic

Not a pipeline metric — this scores the finished film itself. A multimodal LLM judge, blind to which system made which film, rates it 1–5, across our two evaluation studies.

Study 1 — vs. S-Agent & UniVA (main benchmark)

FrameWorkers (ours)

4.32

UniVA

4.27

S-Agent (single-agent)

4.08

Study 2 — vs. Anim-Director & MovieAgent (story-level benchmark)

FrameWorkers (ours)

4.04

Anim-Director

2.88

MovieAgent

1.80

Study 1 also highest on frame-level CLIP video-text alignment: 25.15 (ours) vs. 24.94 (S-Agent) and 24.66 (UniVA).

End-to-end video quality — human evaluation

The same finished films, this time rated blind by human evaluators (1–5) on three axes, across our two evaluation studies.

Study 1 — vs. S-Agent & UniVA (main benchmark)

Instruction following

FrameWorkers (ours)

3.01

UniVA

2.82

S-Agent (single-agent)

2.63

Temporal consistency

FrameWorkers (ours)

3.19

UniVA

2.90

S-Agent (single-agent)

2.67

Realism

FrameWorkers (ours)

3.00

UniVA

2.86

S-Agent (single-agent)

2.68

Highest on every dimension and on 8 of 9 individual sub-questions — UniVA edges ahead only on two instruction-following sub-questions.

Study 2 — vs. Anim-Director & MovieAgent (story-level benchmark)

Instruction following

FrameWorkers (ours)

3.50

Anim-Director

2.79

MovieAgent

2.02

Temporal consistency

FrameWorkers (ours)

3.09

Anim-Director

2.49

MovieAgent

2.28

Realism

FrameWorkers (ours)

3.13

Anim-Director

2.73

MovieAgent

2.72

Overall

FrameWorkers (ours)

3.21

Anim-Director

2.65

MovieAgent

2.38

From our paper, FrameWorkers: A Dynamic Multi-Agent Framework for Automated AIGC Video Production — currently under review.

The team

FrameWorkers is a research project from INSAIT.

Zhendong Li¹, Lei Sun¹, Letian Shi¹, Deheng Zhang¹, Ruibo Ming¹, Mengshun Hu¹, Dannong Xu¹, Jian Wang², Danda Pani Paudel¹, Luc Van Gool¹, Jinjin Gu¹

¹INSAIT, Sofia University "St. Kliment Ohridski", Sofia, Bulgaria ²Snap Inc., New York, USA