Most creative software still expects a human to sit between the idea and the finished file. You might describe a video edit to an AI, but the actual work of trimming clips, adjusting layers, and exporting frames usually falls back to you. That gap exists because media editing is not a single prompt-and-response task. It is a long chain of dependent decisions where step three only makes sense if step two actually changed the timeline. A recently shared project tackles this exact friction by wiring Claude Code into a stateful video editing pipeline powered by the Gemini Interactions API. The result is a working demonstration of how to get an AI agent to truly direct a creative workflow rather than simply suggest one.
A Director and an Editor
The architecture is intentionally split. Claude Code operates as the director, handling high-level planning, interpreting vague creative briefs, and deciding what needs to happen next. It breaks a request like “cut the dead air and add a title card” into discrete tasks, then monitors whether each task succeeds before moving on.
The Gemini Interactions API handles the specialized editing logic. Instead of forcing a generalist model to simulate a video editor, this setup uses Google’s API as the hands-on operator that executes cuts, evaluates timeline state, and reports back with concrete results. The system does not collapse both jobs into a single model. It keeps the reasoning layer separate from the tool-use layer, which means each part can focus on what it does best.
This division mirrors how actual post-production teams function. A director knows the story and makes the call. The editor knows the software and manipulates the pixels. When an agent tries to do both inside one context window, it often trips over syntax or forgets which clip is on which track. Splitting the load fixes that.
Why Memory Changes Everything
Video editing is inherently stateful. If you shorten a clip by four seconds, every subsequent transition, audio cue, and subtitle placement must shift to match. Most AI agents struggle here because they treat each step as an isolated query. They might recommend a cut in one response, then hallucinate a different timeline in the next, or suggest an effect for a segment that no longer exists.
The Gemini Interactions API is built to maintain state across these operations. It tracks the actual condition of the project as the agent works. That means the system knows whether an export failed, whether a clip has already been processed, or whether a color grade has been applied. When Claude issues the next instruction, it is working against the real current state of the timeline, not a guess.
For anyone who has watched an AI confidently suggest a “simple” five-step fix that completely ignores the previous four steps, the value of persistent state is obvious. Creative tasks degrade quickly without memory. A stateful backend turns a chatbot into a participant that can actually finish a job.
What the Workflow Looks Like
Picture a practical sequence. You feed the system several raw clips and ask for a finished social-media cut. Claude Code first evaluates the request. It might decide the footage needs stabilization, then a voice-over extraction, then subtitles, then a vertical crop. It structures these as a plan and begins calling the Gemini Interactions API to execute each item in order.
Gemini performs the media operations and returns structured feedback. Maybe the stabilization succeeded but the audio separation found overlapping dialogue that makes subtitles unreliable. Claude receives that update, revises the plan, and asks Gemini to try a different approach, such as identifying speaker segments before generating text overlays. Because the API holds the state, it can confirm that the subtitle track aligns with the newly stabilized video, not the original shaky footage.
This loop continues until the checklist is complete. The system creates a genuine workflow for complex video tasks instead of a one-off script generation. If a render fails because a codec setting is incompatible, the error feeds back up to Claude, which can adjust parameters and retry. The agent does not abandon the project after the first hurdle.
Using Different Models for Different Strengths
The project also illustrates a broader design pattern in AI engineering: stop trying to make one model do it all. Claude Code excels at reasoning through ambiguous instructions, managing branching logic, and maintaining conversational context over a long session. The Gemini Interactions API, particularly in its interaction with rich media and tool use, brings deep multimodal capabilities and stateful execution. By wiring them together, you route around the limits of each.
A reasoning model that has never touched a nonlinear editor can still direct a great cut if it has access to an execution layer that understands codecs, keyframes, and track hierarchies. Conversely, a media-savvy API does not need to parse abstract creative notes if a planner model has already translated them into concrete steps. The strengths of one fix the weaknesses of the other.
This is not theoretical. The setup explicitly demonstrates how different AI models work together to handle a category of work, creative video editing, that single-model agents often fail to finish. For developers building agentic systems, the lesson is hard to ignore. Stop asking your orchestration layer to be your specialist, and stop asking your specialist to be your strategist.
What Builders Should Take Away
You do not need to be running a video studio to find this pattern useful. Any domain that requires multi-step work over a changing environment, CAD workflows, audio engineering, data visualization, or scientific computing, can borrow the same structure. One model acts as the persistent project manager. A specialized API or tool handles the stateful operations inside the domain-specific software.
The developer’s job shifts from writing giant prompts that pray the model remembers everything, to designing clean handoffs between reasoning and execution. State management becomes the critical piece. If your agent cannot see what changed after its last action, it cannot reliably act again.
You can read the full breakdown of how the integration works, including the specific API interactions and project structure, in the detailed post on dev.to.
The Real Takeaway
Solving complex creative work with AI does not require waiting for a single, perfect model that can plan, remember, and execute everything at once. It requires giving the agent a memory that survives between steps and a specialist it can actually delegate to. Let the thinker think. Let the editor edit.