Prompting a Joint Audio-Video Model

One prompt, two readers

Everything unusual about prompting this model follows from one fact established in f1: the audio and video streams receive different context embeddings from the same prompt.

You write one paragraph. Two conditioning signals come out of it. The picture stream reads it; the sound stream reads it too, differently. Nobody hands you separate fields — the documented request body has one prompt, up to 5000 characters.

The practical consequence is blunt. A prompt with no sound in it is still generating sound, because generate_audio defaults to true and the audio stream is reading whatever you wrote. If your prompt is a purely visual description, the audio stream is inferring a soundtrack from visual language. It will produce something. You just did not choose it.

So the discipline is: write for both readers, in the same prose, deliberately.

Scene, camera, mood

A structure that holds up across shots:

Scene — the subject and what it is doing, and where. Concrete nouns and verbs. "A woman in a wet raincoat pushes open a café door" beats "a cinematic urban moment."

Camera — framing and distance in words, motion in the parameter. Prose handles what prose is good at: close on her hands, wide from across the street. The camera_motion enum handles movement, and it is a structured control with eight values, which is far more reliable than hoping an adjective lands.

Mood — light, texture, weather, time of day, and at least one sound cue. This is the layer that feeds both readers at once. "Late afternoon, low sun through rain-streaked glass, the hiss of traffic outside and a milk steamer behind the counter" gives the video stream light and the audio stream a specific mix — from one sentence.

That last example illustrates the habit worth building: name the sounds you want, in the same breath as the light. Not as a separate sound-design paragraph tacked on the end, but woven into the description of the place. You are describing an environment, and environments have both.

The eight camera motions

The complete enum, and nothing else is in it:

dolly_in · dolly_out · dolly_left · dolly_right · jib_up · jib_down · static · focus_shift

Look at the shape of that list rather than memorising it. Seven of the eight are camera positions: in and out along the depth axis, left and right along the lateral axis, up and down along the vertical axis, or nowhere at all. The eighth, focus_shift, does not move the camera — it changes what is sharp.

There is no rotation in this enum and no lens change. No pan, no tilt, no orbit, no zoom, no handheld. Every available motion is a translation of the camera body through space, plus focus. That is genuinely useful to know: it tells you the enum's mental model is a camera on a dolly or a jib, and if your shot idea depends on the camera pivoting in place or the lens length changing, no parameter will give it to you.

How to use each one as a tool:

• `dolly_in` — commitment. It says this matters, look closer. Strong on a face or an object at the moment it becomes important. Overused it reads as pushy.
• `dolly_out` — revelation or abandonment. Pull back and the context arrives; pull back at the end of a beat and it reads as departure.
• `dolly_left` / `dolly_right` — lateral travel. Reveals depth by parallax, because foreground and background slide against each other. The workhorse for establishing a space.
• `jib_up` — scale and conclusion. Rising off a subject is the oldest ending in cinema.
• `jib_down` — arrival, descent into a scene. Reads as entering rather than surveying.
• `static` — the one people underrate. For dialogue and performance, static is usually correct: nothing in the frame is competing with the face, and every bit of generation effort goes to the subject rather than to maintaining a move.
• `focus_shift` — directs the eye without moving the camera. The precision instrument of the set: shift attention between two things in the same frame instead of cutting.

Make the prompt agree with the parameter. If you set dolly_in while your prose describes a sweeping aerial, you have given the model two incompatible instructions and you will get an average of them.

Turning the sound off

generate_audio: false is right in three situations:

1. You are scoring in post. A composer, a licensed track, a fixed mix. Generated audio you will discard is a distraction in review.
2. The delivery surface is silent. Autoplay-muted social placements and background loops.
3. The sound is coming from somewhere else. Dialogue driven by an existing audio track is the audio-to-video endpoint, which is Pro-only. Synchronised foley over a muted clip is the video-to-audio capability, delivered as a foley LoRA rather than a cloud endpoint. Both are covered in the tracks. Generating base audio you intend to replace is wasted attention.

There is no listed price difference. Turn it off for a reason, not for a discount.

What prompting cannot do

Two honest boundaries.

Negative prompts are a local capability. The official local pipeline signature takes a negative_prompt — the published example passes "worst quality, low quality, blurry, distorted". The documented cloud request bodies for text-to-video and image-to-video list prompt, model, duration, resolution, plus optional fps, generate_audio, and camera_motion. No negative prompt. On the API you steer with the positive prompt; do not port a local negative-prompt workflow and assume it transfers. (Related: distilled checkpoints run at CFG=1, so guidance-scale technique from other models does not carry over either.)

Consistency is not a prompt problem. You cannot write your way to the same face across twelve shots. The mechanisms for that are adapters, and they are in the tracks.