How to Prompt AI With a Video
A model does not watch your video, it samples frames from it. Knowing the sampling rate tells you which questions it can actually answer.
To prompt AI with a video, you need to know three things about what the model receives: it samples still frames rather than watching continuously, it usually gets the audio track as well, and everything between the sampled frames is gone. Google's API samples at one frame per second by default. So a question like "what happened at 00:42" is answerable, and "did his hand touch the glass before it fell" often is not, because the moment may have fallen between two frames. Write your prompt around that, and video becomes one of the more reliable inputs you can give a model.
What the model gets when you prompt AI with a video
A video is not streamed to the model. It is decomposed, and the shape of that decomposition is public.
According to Google's video understanding documentation, each second of video becomes roughly 100 tokens at low media resolution or about 300 at high resolution. That breaks down as one sampled frame per second, costing 66 tokens at low resolution or 258 otherwise, plus 32 tokens per second for the audio track, plus metadata. On a model with a one million token context, that supports videos up to three hours at low resolution or one hour at high.
Two consequences fall out of this immediately.
The first is cost and context. Ten minutes of video at high resolution is roughly 180,000 tokens. That is not a casual attachment. Clip before you upload.
The second is the sampling gap. At one frame per second, a quick cut, a fast gesture or a frame of on-screen text that flashes for half a second may simply not exist in what the model sees. The documentation says as much, noting that the default rate may miss details in fast-motion or quick-cut video.
Say where to look, in MM:SS
The single highest-leverage habit is timestamping your question. The API expects the MM:SS format, and prompts that use it get noticeably more grounded answers than prompts that do not.
Instead of:
What does he say about pricing?
Write:
Between 02:10 and 03:40, summarise what the speaker says about pricing. Quote the exact figures. If a figure is spoken but unclear, say so rather than guessing.
The timestamp does two jobs. It narrows the search, and it gives you something to verify against. An answer tied to 02:48 can be checked in eight seconds. An answer with no anchor cannot be checked at all, which is the same problem that makes getting a model to flag what it is unsure about worth building into the prompt.
Clip and slow the sampling before you ask
If the detail you need is finer than one frame per second, change the sampling rather than rewording the prompt. The processing configuration accepts an fps value, and start_offset and end_offset to analyse one segment.
# Sample one frame every two seconds across a 90 second window
processing = {
"fps": 0.5,
"start_offset": "60s",
"end_offset": "150s",
}Going the other way matters more often. A higher fps on a short clip is how you catch fast motion, and restricting the window is how you afford it. One caveat worth reading the docs for: these customisation options apply in static processing mode and not to agentic processing.
Ask questions video can answer
Video answers some question shapes well and others badly. Sorting your question into the right shape before you write it saves a round trip.
Question type | Reliability | Why |
|---|---|---|
What is said, and when | High | Audio is transcribed at a steady 32 tokens per second, independent of frame sampling |
What is visible in a scene | High | One frame per second is plenty for a static scene |
On-screen text that stays up | High | Readable in any sampled frame |
Order of slow events | Moderate | Works when events are seconds apart, not frames apart |
Exact moment of a fast action | Low | The frame may not have been sampled |
Counting fast repetitions | Low | Same reason, and counting is already a weak spot |
Reading a flashed caption | Low | May fall entirely between frames |
When you need something from the bottom half of that table, raise the frame rate on a narrow window and ask again.
A prompt structure that works
Four parts, in this order. The structure is close to the one that works for prompting with a screenshot, with timing added.
What the video is. One line. "A 12 minute screen recording of a checkout flow in a web app."
The window. "Focus on 04:30 to 06:00."
The question, one job at a time. "List every form field the user fills, in order, with the timestamp each one is first clicked."
The uncertainty instruction. "If you cannot see a field clearly, write unclear rather than inferring it."
Here it is assembled:
This is a 12 minute screen recording of a checkout flow in a web app. Focus on 04:30 to 06:00. List every form field the user fills in, in order, with the timestamp each is first clicked. If a field label is not legible in the frame, write unclear instead of guessing it.
That prompt produces a checkable list. The vaguer version, "tell me about this checkout flow", produces a summary you have to watch the video to trust, which defeats the point.
Two mistakes worth naming
Treating the transcript and the visuals as one source. The model receives both, but they are not equally reliable at a given second. Audio runs continuously. Visuals are a sample. When a speaker says one thing and a slide shows another, ask about them separately.
Uploading the whole recording out of habit. A two hour webinar at high resolution is most of a million token context before you have asked anything. Clip to the part you care about. The same instinct that makes people paste a whole PDF rather than the relevant pages causes most of the trouble in summarising long documents accurately.
Frequently asked questions
Does the model hear the audio in my video?
Yes, in Google's API the audio track is processed alongside the frames at 32 tokens per second, and the documentation notes audio is processed at 1Kbps single channel. The transcript side of a video prompt is usually its most reliable part.
How long a video can I send?
With a one million token context window, Google documents up to three hours at low media resolution or one hour at high. Cost, not the limit, is usually what stops you first.
Why does the model miss something I can clearly see?
Most often because the frame containing it was never sampled. At the default one frame per second, anything shorter than a second can fall between frames. Raise the fps for that segment and ask again.
Should I use low or high media resolution?
Low resolution for talking-head video, screen recordings of mostly static pages, and anything where you mainly want the audio. High resolution when you need to read small on-screen text or judge fine visual detail. The cost difference is roughly threefold.
Is a video better than a screenshot for a UI question?
Only if the answer involves change over time. For "why does this layout look wrong", a screenshot is cheaper and sharper. For "what happens when I click submit", use the video. Multimodal models take both, and the right choice is whichever carries less noise.
How did this land?
About the author
Developer Advocate
Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.
