Programu nyingi za ubunifu bado zinahitaji binadamu kuwa kati ya wazo na faili iliyokamilika. Unaweza kuelezea uhariri wa video kwa AI, lakini kazi halisi ya kukata vipande, kurekebisha tabaka, na kuhamisha fremu kwa kawaida inabaki kwako. Pengo hilo lipo kwa sababu uhariri wa vyombo vya habari si kazi ya ombi na jibu moja tu. Ni mnyororo mrefu wa maamuzi yanayotegemeana ambapo hatua ya tatu ina maana tu ikiwa hatua ya pili ilibadilisha timeline kweli. Mradi ulioshirikishwa hivi karibuni unatatua msuguano huu kwa kuunganisha Claude Code kwenye mfumo wa uhariri wa video wenye uwezo wa kuhifadhi hali (stateful) unaoendeshwa na Gemini Interactions API. Matokeo yake ni onyesho linalofanya kazi la jinsi ya kumfanya wakala wa AI aongoze mchakato wa ubunifu badala ya kupendekeza tu mmoja.
Muongozaji na Mhariri
Muundo huu umegawanywa kwa makusudi. Claude Code hufanya kazi kama muongozaji, akishughulikia mipango ya kiwango cha juu, kutafsiri maelekezo ya ubunifu yasiyo wazi, na kuamua nini kinapaswa kufanyika baadaye. Inavunja ombi kama “kata sehemu zisizo na sauti na ongeza kadi ya kichwa cha habari” kuwa kazi mbalimbali, kisha inafuatilia ikiwa kila kazi imefanikiwa kabla ya kuendelea.
Gemini Interactions API inashughulikia mantiki maalum ya uhariri. Badala ya kulazimisha modeli ya jumla kuiga mhariri wa video, mpangilio huu unatumia API ya Google kama mwendeshaji anayefanya kazi kwa mikono anayetekeleza makato, anayechunguza hali ya timeline, na kutoa ripoti yenye matokeo madhubuti. Mfumo huu haujiunganishi kazi zote mbili katika modeli moja. Unatenganisha tabaka la ufikiri na tabaka la matumizi ya zana, kumaanisha kila sehemu inaweza kuzingatia kile inachofanya vizuri zaidi.
Mgawanyo huu unaiga jinsi timu halisi za baada ya uzalishaji (post-production) zinavyofanya kazi. Muongozaji anajua hadithi na hufanya maamuzi. Mhariri anajua programu na hurekebisha picha (pixels). Wakala anapojaribu kufanya yote mawili ndani ya dirisha moja la muktadha (context window), mara nyingi hukwama kwenye kanuni za lugha (syntax) au kusahau ni kipande gani kiko kwenye njia (track) gani. Kugawanya mzigo huo kunatatua tatizo hilo.
Kwa Nini Kumbukumbu Inabadilisha Kitu Kila Kitu
Uhariri wa video kiasili una hali inayobadilika (stateful). Ukipunguza kipande cha video kwa sekunde nne, kila mpito unaofuata, ishara ya sauti, na mpangilio wa maandishi (subtitles) lazima ubadilike ili kuendana. Wakala wengi wa AI wanapata shida hapa kwa sababu wanachukulia kila hatua kama swali linalojitegemea. Wanaweza kupendekeza kata katika jibu moja, kisha kuunda timeline tofauti katika jibu linalofuata, au kupendekeza athari (effect) kwa sehemu ambayo haipo tena.
Gemini Interactions API imeundwa ili kudumisha hali (state) katika shughuli hizi. Inafuatilia hali halisi ya mradi wakati wakala anapofanya kazi. Hiyo inamaanisha mfumo unajua ikiwa uuzaji (export) umefeli, ikiwa kipande tayari kimeshakusudiwa, au ikiwa rangi imeshatumika. Claude anapotoa maelekezo yajayo, anafanya kazi kulingana na hali halisi ya sasa ya timeline, siyo kwa kubahatisha.
Kwa yeyote aliyewahi kuona AI ikipendekeza kwa ujasiri marekebisho "rahisi" ya hatua tano ambayo yanapuuza kabisa hatua nne zilizopita, thamani ya hali endelevu (persistent state) ni dhahiri. Kazi za ubunifu hupoteza ubora haraka bila kumbukumbu. Mfumo wa nyuma wenye uwezo wa kuhifadhi hali (stateful backend) unageuza chatbot kuwa mshiriki anayeweza kumaliza kazi kweli.
Jinsi Mchakato Unavyoonekana
Wazia mfuatano wa vitendo. Unalisha mfumo vipande kadhaa vya video ghafi na kuomba toleo lililokamilika la mitandao ya kijamii. Claude Code kwanza inatathmini ombi. Inaweza kuamua kuwa video inahitaji utulivu (stabilization), kisha uondoaji wa sauti (voice-over extraction), kisha maandishi (subtitles), kisha kukata kwa wima (vertical crop). Inaandaa haya kama mpango na kuanza kuita Gemini Interactions API ili kutekeleza kila kipengele kwa mpangilio.
Gemini inafanya kazi za vyombo vya habari na kutoa mrejesho uliopangwa. Labda utulivu ulifanikiwa lakini utenganishaji wa sauti ulipata mazungumzo yanayoingiliana ambayo hufanya maandishi (subtitles) kutokuwa na uhakika. Claude inapokea taarifa hiyo, inarekebisha mpango, na kuiomba Gemini kujaribu njia tofauti, kama vile kutambua sehemu za mzungumzaji kabla ya kutengeneza maandishi ya juu (text overlays). Kwa sababu API inahifadhi hali (state), inaweza kuthibitisha kuwa njia ya maandishi inaendana na video iliyotulia hivi karibuni, siyo video ya awali inayoyumba.
Mzunguko huu unaendelea hadi orodha ya kazi iishe. Mfumo huu unatengeneza mchakato halisi wa kazi za video tata badala ya kutengeneza skripti ya mara moja tu. Ikiwa render itafeli kwa sababu mpangilio wa codec haukubaliani, hitilafu hiyo inarudi kwa Claude, ambayo inaweza kurekebisha vigezo na kujaribu tena. Wakala haachi mradi baada ya kikwazo cha kwanza.
Kutumia Modeli Tofauti kwa Nguvu Tofauti
The project also illustrates a broader design pattern in AI engineering: stop trying to make one model do it all. Claude Code excels at reasoning through ambiguous instructions, managing branching logic, and maintaining conversational context over a long session. The Gemini Interactions API, particularly in its interaction with rich media and tool use, brings deep multimodal capabilities and stateful execution. By wiring them together, you route around the limits of each.
A reasoning model that has never touched a nonlinear editor can still direct a great cut if it has access to an execution layer that understands codecs, keyframes, and track hierarchies. Conversely, a media-savvy API does not need to parse abstract creative notes if a planner model has already translated them into concrete steps. The strengths of one fix the weaknesses of the other.
This is not theoretical. The setup explicitly demonstrates how different AI models work together to handle a category of work, creative video editing, that single-model agents often fail to finish. For developers building agentic systems, the lesson is hard to ignore. Stop asking your orchestration layer to be your specialist, and stop asking your specialist to be your strategist.
What Builders Should Take Away
You do not need to be running a video studio to find this pattern useful. Any domain that requires multi-step work over a changing environment, CAD workflows, audio engineering, data visualization, or scientific computing, can borrow the same structure. One model acts as the persistent project manager. A specialized API or tool handles the stateful operations inside the domain-specific software.
The developer’s job shifts from writing giant prompts that pray the model remembers everything, to designing clean handoffs between reasoning and execution. State management becomes the critical piece. If your agent cannot see what changed after its last action, it cannot reliably act again.
You can read the full breakdown of how the integration works, including the specific API interactions and project structure, in the detailed post on dev.to.
The Real Takeaway
Solving complex creative work with AI does not require waiting for a single, perfect model that can plan, remember, and execute everything at once. It requires giving the agent a memory that survives between steps and a specialist it can actually delegate to. Let the thinker think. Let the editor edit.