How to Animate a 2D Character from a Voice-over with an AI Agent

The big picture: 8 steps

From voice-over to finished video. You supply the VO, review the result and ask for changes.

You design the character once (step 1). After that, each new video repeats steps 2 to 8. The agent copies the previous video project as a template, so the renderer, captions, effects and mix settings never have to be rebuilt.

Step 0: Install the skill and set up a workspace

1. Install MotionMind SkillFull. After buying, follow the store's install instructions. For Claude Code, skills live in ~/.claude/skills/. Open Claude Code and ask "which skills are available?" to confirm it loaded.

2. Install MotionMind and create a workspace.

doctor checks Node, ffmpeg and the rest. Make sure every line shows ✓ before you continue.

3. Open the Studio in a second terminal and leave it running:

The Studio opens at http://127.0.0.1:4310. This is where every version of your video gets logged, played back and compared.

Step 1: Design the character once, reuse it forever

A consistent series depends on a character built as code, not a flat picture. Desfina is a layered SVG. Every part has its own id (hair-back, head, eye-l, eye-r, brow-l, mouth, arm-r, hand-r), so each one can move on its own: blinking, glancing, nodding, pointing and talking.

Desfina's character sheet: 7 expressions, 5 lip-sync mouth shapes, a colour palette and a profile picture, all from one generator. The labels are in Indonesian.

An example prompt for the character:

Create a flat-vector 2D female character named Desfina for business content. Long black hair, teal blazer, gold accents. Build her as an SVG with separate layers for hair, head, eyes, brows, mouth and arms. Include 7 expressions (happy, confident, chatting, surprised, thinking, wink, laughing) and 5 lip-sync mouth shapes (rest, E/I, A, O/U, smile). Export a character sheet, full-body poses and a profile picture.

Once you've picked your favourite version, ask the agent to record it in the project's TASTE.md: palette, fonts, style, character placement and caption rules. The agent reads this file every time it starts a new video, so the look stays the same even in a fresh chat.

Step 2: Prepare the voice-over

Record your script as a WAV and put it in a topic folder, for example desfina/blockbuster/vo.wav. A few tips:

• 45–75 seconds suits Reels well. The video length follows the VO.
• Keep the voice clean, with no background music. Music and SFX are added later.
• Say names, numbers and years clearly. Those are the moments that get the biggest visuals.

Step 3: Brief the agent

This is the prompt we actually used for every Desfina video, translated from Indonesian:

Now make content that follows this VO, animated like the previous ones. Keep the Desfina hair branding, and fill it with sound effects, transitions, camera moves, keyframes and so on. If you don't know what it's about, research it first, especially when a brand is involved. Feel free to show the people being discussed as animated versions of their faces. Export to the same folder, and don't forget the thumbnail.

The prompt is short because the rules already live in the skill and in TASTE.md. After this brief the agent will:

1. Read the VO and research the topic. For the Blockbuster video it checked the $50 million offer in 2000, the store count at its peak and the year it went bankrupt. Any claim it couldn't verify was reported back.
2. Copy the previous project as a template and draft the scene list.
3. Work through steps 4 to 9 below, then report the result, duration and loudness.

Step 4: Transcript and beats.json

The agent transcribes the VO locally with faster-whisper, so every word gets a start and end time. The transcript is always reviewed. For example, a misheard word was corrected and word onsets that drifted were snapped to the energy of the voice.

From those words the agent builds one central file: plan/beats.json.

The word "menertawakan" ("laughed at") starts at 1.76 s. That same number fires the on-screen text, the camera shake and the laughing SFX.

The principle: picture and sound read the same numbers. If the VO is re-recorded, update beats.json and every animation and sound effect moves with it.

Step 5: Build the scenes

Each scene is one JavaScript file in src/scenes/. Their time ranges overlap slightly so transitions blend: a whip-pan with motion smear, an iris, a colour wave or a diagonal wipe.

The Blockbuster video has 12 scenes. Each file draws one part of the story.

Motion uses keyframes computed from time, never from the previous frame. Here's a snippet from the boardroom scene:

Because every frame depends only on the time t, a re-render always produces exactly the same image. You can also render just a 5-second slice to check one moment.

The people in the story become caricatures

Business stories land better when you can see who's involved. The agent built one caricature generator (src/figure.js) with parameters for hair, glasses, moustache, outfit, pose and expression, drawn in the same line style as Desfina.

Tirto Utomo, Djajadi Djaja, Gunpei Yokoi, the Dassler brothers, John Antioco, and Reed Hastings with Marc Randolph. All from the same generator.

A few rules we stick to:

• No real logos. Brand names are set in our own typography.
• No copyrighted characters. Mario and Donkey Kong only appear as names on generic cards.
• Real people are drawn as respectful caricatures, not mockery.

Step 6: Lip-sync

Desfina's mouth follows the voice. The lip-sync script reads the energy and character of the voice in every frame and picks one of the four mouth shapes from the character sheet. Add a blink about every 3 seconds and small nods while she speaks, and she really looks like she's talking.

Each block is one frame (1/30 s). Its colour shows which mouth shape is used.

Step 7: Review with a contact sheet

Before a full render, the agent makes a contact sheet: dozens of small frames in one image. It's much faster than rewatching the video, and layout problems jump out immediately.

Contact sheet of the first 23 seconds, one frame every 0.4 seconds.

Problems the contact sheets caught in this series:

• The "$50 JUTA" text covered Desfina's face, so it moved up.
• The people in the boardroom were hidden behind the table, so they were raised.
• Small objects like a mailbox and a garage were too small on a phone screen, so they were enlarged.
• A steering wheel had too little contrast with the background, so its colour changed.

Step 8: Sound design and mix

This is what makes the video feel expensive. Every important movement has a sound: a whoosh on transitions, a pop when a label appears, stamps, coins, cartoon laughter, neon hum, even a passing plane. The Blockbuster video uses 249 SFX cues, a mix of library sounds and synthesized sounds made in code.

Each yellow tick is one SFX cue pinned to an animation event. The final mix still leaves room for the narration.

The mix rules:

• The voice always comes first. SFX automatically dip by up to 4 dB under speech (ducking).
• Final loudness is −14 LUFS with a true peak of −1.5 dBTP, a safe target for Reels, TikTok and YouTube Shorts.
• No music unless it's requested.

Step 9: Render and thumbnail

Rendering uses Playwright to draw every frame in Chromium, then ffmpeg joins them with the audio into a 1080×1920, 30 fps MP4. Each project's README.md records the exact commands, so you can re-render at any time:

The thumbnail is never a frame grabbed from the video. We learned that from our own revisions: a video frame looks flat as a cover. Each thumbnail is designed as its own poster, with cut-out stickers of the people involved, a 3D headline, halftone or retro-newspaper texture, and Desfina peeking in from a corner.

12 poster thumbnails from the Desfina series. Same character and style, a different concept every time.

Step 10: Log versions in MotionMind Studio

Once there's a v2, a v3 and more, you need to know which one is current and what changed. MotionMind records every render as a frozen version:

In the Studio (motionmind ui) you can play each version, see its contact sheet and loudness, and compare versions side by side. Promote an approved version to production with motionmind promote.

How to ask for changes

Because everything is code, revisions touch only the part you name, without starting over. Requests that work well:

• "At 12–14 s the 9,000 counter is hard to read. Make it bigger and add an outline."
• "The laughing SFX in the boardroom is too quiet. Bring it up to the level of the voice."
• "Redo the thumbnail. Don't use a video frame; design a more creative poster."
• "Save it as v2 and don't overwrite v1."

The agent always saves a revision as a new version, so a version you've approved is never overwritten.

Common mistakes

• Rendering the full video straight away. Render a short slice and check a contact sheet first. It saves a lot of time.
• Using real brand logos. Use your own typography. It's safer and keeps the look consistent.
• Trusting the loudness number alone. −14 LUFS doesn't guarantee every SFX is audible. Listen to the result yourself before you post it.
• Not checking facts in the VO. Have the agent research first. In this series it flagged several numbers in the VO that differed from the latest official figures.
• No TASTE.md. Without a style record, every new chat can drift to a different look.

Start your own character series

Setting up the character and the first video takes a few hours. After that, each new video is one VO and one prompt. Everything in this article was made with the same tools:

• MotionMind SkillFull skill: buy on Agensi or get the Indonesian edition on Yapp
• MotionMind CLI & Studio: free, npm i -g motionmind. The full guides are on the Learn page.

Got a character or a business story you want to animate? Make your first video, then compare every version in the Studio.