# AI Video Prompting Knowledge Base *A shareable knowledge document. Paste it into any Claude project, custom GPT or system prompt to give your assistant expert-level AI video prompting guidance.* --- ## OVERVIEW This knowledge base covers best practices for prompting today's AI video models: **Seedance 2.0 / 2.5, Kling 3.0, Google Veo 3.1, Gemini Omni Flash and Grok Imagine**. It's built from creator community insights and the platforms' own guides. **Sora is gone.** OpenAI shut down the Sora app and website on April 26, 2026, and the Sora API on September 24, 2026. Many creators came from Sora, and the core lesson they learned still applies: **today's models won't infer what you mean.** Emotion, pacing, camera movement and shot structure all need to be spelled out. The storyboard format Sora used internally is still one of the best ways to do that. --- ## THE STORYBOARD FORMAT (Universal Template) The most effective structure for almost any video model is a shot-by-shot storyboard. Use it to "upscale" a basic idea before you send it to a model. ### Format for each shot: ``` Shot [N] ([duration]s): [Scene description — what is happening, who is in it, where they are] [Camera behavior — movement, angle, framing, start and end position] [Environmental details — lighting, weather, atmosphere] [Character details — expression, posture, action] [Sound — ambience, effects, music, dialogue in quotes] * Camera shot: [shot type, e.g. "long continuous handheld POV, backward tracking"] * Depth of field: [e.g. "moderate shallow focus — subject sharp, background softly blurred"] * Mood: [single descriptor, e.g. "uncanny and ominous"] * Video quality: high ``` ### Example of a fully expanded shot: > **Shot 1 (4.0s):** > Long continuous shaky handheld iPhone POV pulling backwards down the middle of an upscale suburban street on a dark cloudy day. Fresh driveways, classic mailboxes, perfectly groomed dark green lawns and hedge bushes line both sides. Gusty wind shakes healthy suburban oak trees. In the distance, a single fluffy black angry cat rides steadily toward camera on a black Roomba with a red light ring. Ultra-realistic, natural lighting, cinematic handheld motion. Sound: wind gusts, distant dog bark, the low whir of the Roomba growing louder. > > * Camera shot: long continuous handheld POV, centered backward tracking > * Depth of field: moderate shallow focus (foreground street and distant subject emphasized, homes softly blurred) > * Mood: uncanny and ominous > * Video quality: high ### Key principles of the storyboard format: - A 10-second cinematic video usually works best as **3–5 shots**. Fewer, longer shots are more stable; more shots feel more edited. - Give each shot an explicit **start state**, **progression** and **end state**. - If you want a naturalistic look, say **"No dialogue"** and **"Unedited raw footage."** - Models with **native multi-shot** support (Kling 3.0, Seedance 2.5) can take several labeled shots in one generation. With others, generate one shot at a time and edit them together. --- ## MODEL-SPECIFIC PROMPTING RULES *Current as of October 2026. Features and limits change often, so check each platform's latest docs.* ### Seedance 2.0 / 2.5 (ByteDance: Dreamina / CapCut, Jimeng, BytePlus) - **Hyper-literal:** it reads prompts exactly as written. A typo or a loose comparison produces a literal result ("boulder-sized backpack" renders an actual boulder). - **Spellcheck everything** before you submit. - **Camera language is required:** "dolly out," "snap zoom," "handheld POV," "tracking shot," "dutch angle." - **Define the endpoint:** describe where the camera *finishes*, not just where it starts ("dolly out until we can see the full skyline"). - **Simple prompts also work:** clean, concise prompts often beat over-engineered ones. - **Multiple references:** image, video and audio references can be combined. Give each one a single job ("Image 1 controls her face only"). - **2.5 adds timeline prompting:** map your assets, write a one-sentence brief, then give a continuous second-by-second timeline and global constraints. It also adds instruction-based editing, extension, and first/last-frame generation that lands much closer to the requested end frame. See the *Seedance Prompting Guide* for templates. - Commonly recommended for **multi-shot storytelling** and projects with **many reference assets**. ### Kling 3.0 (Kuaishou) - **Native multi-shot:** a single generation can hold a short storyboard (guides report up to about 5–6 shots in roughly 15 seconds). Label each shot and give it its own framing, subject and motion. - **Elements:** reusable characters, objects and scenes built from several reference images (reported as 2–4). Build them once and reuse them for consistency. - **Native audio with lip sync**, in several languages. - **Motion Control:** feed a reference video (a dance, a stunt, your own acting) plus a character image to map your character onto that exact motion. - Understands cinematic shorthand: profile shot, macro close-up, POV, shot-reverse-shot dialogue. - **Kling 3.0 Turbo** is a faster, cheaper variant (3–15 second clips). Use it for drafts. - Strong for **motion-heavy** work and often the most credit-efficient option. ### Google Veo 3.1 (Flow, Gemini app, Gemini API / Vertex AI) - Prompt formula from Google's guidance: **cinematography + subject + action + context + style and ambiance**, with audio given as its own instruction. - **Veo always generates sound**, so write the sound you want, or you'll get whatever it guesses. - **Ingredients to Video:** up to 3 reference images (character, object or scene) for consistency across shots. - **Frames to Video:** give it a start image and an end image, and it generates the motion between them. Keep the prompt short, because the images carry the look. - **Extend:** clips are about 8 seconds, so chain or extend for longer shots, reusing the last frame as the next start. - Keep each clip to **one shot, one main action**. - Community tip: if motion feels "floaty," add **"dynamic cinematic cuts throughout"** or describe faster, more physical camera work. - Best known for **cinematic realism and native audio**. Check whether your Google One / Workspace plan already includes Flow before paying elsewhere. ### Gemini Omni Flash (Google, launched at I/O in May 2026) - A multimodal model that takes text, images, audio and video in, produces video out, and lets you **edit conversationally** ("make it night," "swap the jacket to red"). - Clips up to about **10 seconds at 720p**. **Omni 1.1 Flash** adds first/last-frame control, 4K upscaling, fast **360p draft mode** and scene extension. - Testers report its raw generation trails Seedance and Kling, but its editing is stronger. Use it to *fix and iterate* as much as to generate. - Every clip carries Google's SynthID watermark. ### Grok Imagine (xAI, Imagine 1.5) - Generates the **picture and sound in one pass**, including lip-synced dialogue. Put spoken lines in **double quotes**. - Specify ambience, key sound effects and **music or silence**. Left unspecified, it may add music you didn't want. - Community tip: put audio and music instructions **at the start** of the prompt to reduce the hiss artifact. - **Extend from Frame:** pick the exact frame to continue from, then write a short "what happens next" prompt. Shorter extensions and slower motion give cleaner seams, and quality drops after many chained extensions. - Good for **high-volume practice and iteration** at low cost. - Good for **reverse-engineering trends:** give a trending clip to Grok or Gemini and ask it to describe the motion and camera pattern as a reusable prompt template. --- ## PROMPTING PRINCIPLES (APPLIES TO ALL MODELS) ### 1. Over-explain character emotions Models don't infer emotion from context. State it physically: - Facial expression ("eyes narrowed, jaw clenched, lips pressed together") - Body language ("shoulders tense, leaning slightly forward, fists at sides") - Energy and tone ("anxious but trying to appear calm") - Don't write "she's nervous"; show it in the physical description. ### 2. Explicit camera direction Always include: - **Camera type:** handheld, steadicam, drone, dolly, crane, static - **Movement:** tracking, push-in, pull-back, pan, tilt, orbit - **Speed:** slow drift, fast snap, gradual reveal - **Start and end position:** "camera begins at eye level behind the subject, slowly pulling back and rising to a bird's-eye view by the end of the shot" ### 3. The sniper-then-shotgun method 1. **Sniper phase:** generate one video at a time, change one element of the prompt between attempts, and see what changed. 2. **Shotgun phase:** once a prompt is a "winner," generate 3–5 versions in parallel and pick the best. - Don't skip straight to shotgun, or you'll burn credits without learning what works. ### 4. Draft cheap, finish expensive Most platforms now have a fast or low-resolution mode: Kling 3.0 Turbo, Seedance Fast, Veo Fast / Lite, Gemini Omni's 360p drafts. Work out the timing, blocking and camera there, then re-run the winner at full quality. ### 5. Describe sound explicitly Nearly every major model now generates audio natively, so direct it: - Name specific sounds: "the dry crack of cast iron hitting concrete," "a single sharp 'mrow!'" - Describe music by feel: "urgent low strings building to a brass crescendo," or say **"no music."** - Describe the ambience: "distant wind through dry leaves, no traffic, otherwise silent." - Put dialogue in quotes and say who speaks it. ### 6. Respect prompt-length limits If a platform caps prompt length, cut repetition and adjectives before you cut camera, action or sound. Denser, more specific prompts beat longer ones. --- ## CHARACTER CONSISTENCY ACROSS GENERATIONS ### Build a character asset folder Before you generate any videos with a character, prepare: - **Images from all angles:** front, 3/4 front, profile, 3/4 back, back - **A strong video reference:** even 5–10 seconds of the character in motion - **Voice:** a consistent voice (a voice-library voice, or a cloned voice you have rights to) - **A "character bible":** a written description of personality, mannerisms, speech patterns and physical traits Load these into the platform's reusable-character feature where there is one (Kling **Elements**, Veo **Ingredients**, Seedance multi-reference) and attach them on every generation. ### Anchor your sets To keep scenes in a consistent location: 1. Generate a reference image of the room or location (Seedream / Dreamina, Nano Banana, ChatGPT and others all work). 2. Use that image as the base for every character image intended for that scene. 3. Feed those anchored images into every video shot set in that location. - Result: every clip feels like the same world, the same light, the same space. ### First-and-last-frame control and stitching Veo 3.1, Kling 3.0, Seedance and Gemini Omni 1.1 all accept a start image and an end image. 1. Make the **last frame by editing the first** (not regenerating it), so identity, wardrobe and lighting match. 2. Use the **final frame** of each clip as the **first frame** of the next. 3. Stitch the clips in your editor (Premiere, DaVinci Resolve, CapCut). 4. **Trim exactly one frame** at each join. Otherwise the repeated frame shows up as a visible freeze. See *Reference Images for AI Video: First Frames, Last Frames and Character References* for full templates. ### Infinite loop technique For a seamlessly looping social video, set the **very first frame** of the sequence as the **last frame** of the final clip. The video ends exactly where it began. --- ## REAL PEOPLE AND LIKENESS Major platforms restrict generating real, identifiable people, especially public figures, and that's deliberate. Use your own likeness, a performer who has given consent, or an original character. Don't try to trick a platform's face or likeness filters: it breaks their terms, can get your account banned, and is how non-consensual deepfakes get made. If your character needs a recognizable face, build an original one and keep it consistent with the asset folder above. --- ## AUDIO, DIALOGUE AND LIP SYNC ### Use native audio first Kling 3.0, Veo 3.1, Seedance, Grok Imagine and Gemini Omni generate synced dialogue and sound in the same pass. Write the dialogue in quotes, name the speaker, and describe the delivery ("whispered, hesitant"). ### Lip-sync tools as a fallback When native dialogue isn't good enough, generate the voice separately (for example in ElevenLabs) and run the clip through a dedicated lip-sync tool. Kling and several third-party tools offer one. ### Act it yourself (advanced) 1. Film yourself acting out the scene on your phone. 2. Convert your voice to the character's voice with a voice-changer tool, using a voice you have rights to. 3. Run the performance through **Kling Motion Control** with a photo of your character. 4. Result: your character performs the scene with your timing and delivery. 5. Keep that clip as a **permanent reference video** for future generations. ### Narration as a workaround If dialogue is too difficult, use an **internal monologue** or **narrator voiceover**. It's a legitimate cinematic technique (documentary, film noir, coming-of-age films) and removes the need for lip sync entirely. --- ## PROMPT UPSCALING (BUILD YOUR OWN PROMPT WRITER) Use this instruction to set up a Claude project or custom GPT that upgrades basic prompts automatically: > **System prompt for a Prompt Upscaler:** > "You are an AI video prompt specialist. When given a basic video idea, ask which model it is for (Seedance, Kling, Veo, Gemini Omni or Grok Imagine) if it isn't stated, then expand it into a storyboard. For each shot give: shot number and duration (3–5 shots for a 10-second video), a detailed scene description, camera type and movement with start and end position, character emotion shown through physical description, environment and lighting with a named light source, and a sound line covering ambience, effects, music or 'no music', and any dialogue in quotes. End each shot with four tags: Camera shot, Depth of field, Mood, Video quality. Adapt to the model: one shot and one main action per clip for Veo; labeled multi-shot for Kling 3.0; asset mapping plus a second-by-second timeline for Seedance 2.5; audio first for Grok Imagine. Describe in full sensory detail, and don't summarize." --- ## QUICK REFERENCE: CAMERA TERMINOLOGY GLOSSARY | Term | Meaning | |---|---| | Dolly in / push in | Camera moves forward toward the subject | | Dolly out / pull back | Camera moves backward away from the subject | | Pan left / pan right | Camera rotates horizontally on a fixed axis | | Tilt up / tilt down | Camera rotates vertically on a fixed axis | | Whip pan | Very fast pan that blurs, often used as a transition | | Tracking shot | Camera follows a moving subject | | Orbit / arc | Camera circles around the subject | | Crane / jib | Camera rises or descends smoothly through space | | Handheld POV | Unsteady, first-person perspective with natural shake | | Steadicam | Smooth, gliding movement without shake | | Snap zoom | Fast, sudden zoom in or out | | Dutch angle | Camera tilted diagonally, which creates unease | | Establishing shot | Wide opening shot that shows where we are | | Over-the-shoulder (OTS) | Framed from behind one character, looking at another | | Shot-reverse-shot | Alternating angles between two characters in conversation | | Bird's-eye / overhead | Looking straight down from above | | Worm's-eye | Looking straight up from ground level | | Rack focus | Focus shifts from foreground to background (or the reverse) | | Shallow depth of field | Subject sharp, background blurred | | Deep depth of field | Foreground and background both in focus | --- ## SOURCES Community sources: - Topher's Strategy Session 7 community transcript and its "50 Pro Tips" document - A crowdsourced platform and model survey of 30+ active creators (April 2026) - Sora's internal storyboard prompt format - Community Discord insights and creator field reports Platform and news sources (October 2026 update): - [What to know about the Sora discontinuation — OpenAI Help Center](https://help.openai.com/en/articles/20001152-what-to-know-about-the-sora-discontinuation) - [Introducing Veo 3.1 and new creative capabilities in the Gemini API — Google Developers Blog](https://developers.googleblog.com/introducing-veo-3-1-and-new-creative-capabilities-in-the-gemini-api/) - [Bringing new Veo 3.1 updates into Flow — Google](https://blog.google/innovation-and-ai/products/veo-updates-flow/) - [Google launches Gemini Omni Flash — The Next Web](https://thenextweb.com/news/google-gemini-omni-flash-video-model-io-2026) - [Gemini Omni 1.1 Flash announcement — Google AI](https://x.com/GoogleAI/status/2093008998987403303) - [Kling VIDEO 3.0 Model Guide — Kling AI](https://kling.ai/quickstart/klingai-video-3-model-user-guide) - [Kling 3.0 Prompting Guide — fal](https://blog.fal.ai/kling-3-0-prompting-guide/) - [How to use Grok Imagine — fal](https://fal.ai/learn/tools/how-to-use-grok-imagine) *Last updated: October 2026*