How to Prompt AI With a Screenshot
A screenshot answers "what does it look like" and nothing else. Here is what the model can and cannot see, and the caption structure that fixes most bad results.
Pasting a screenshot works better than describing a screen, but only if you say what you want done with it. The image tells the model what things look like. It does not tell the model what is broken, what should happen instead, or which of the fourteen elements on screen you are pointing at. Almost every bad result from a screenshot prompt comes from assuming the picture carries intent.
The fix takes one extra sentence. Here is the structure, and the specific things models cannot see no matter how sharp the image is.
What the model can read
Modern vision-capable models handle rendered screens well. Reliably readable:
Text that is rendered at a normal size, including labels, buttons, and body copy
Layout and spatial relationships, what is above, beside, or inside what
Colours, spacing, alignment, and obvious visual inconsistency
Table and form structure, including which values sit in which column
Error messages and stack traces, as long as they are not truncated by the viewport
If you want the underlying detail on how images and text get processed together, what is multimodal AI covers the mechanism.
What the model cannot see, and will guess about
This is the list worth internalising, because the model will not tell you it is guessing.
Anything off screen. Scrolled-away content does not exist. Neither does the rest of a long error log.
State that requires interaction. Hover styles, focus rings, open dropdowns, tooltips, disabled states, whatever the second tab contains.
What happened before. A screenshot of a failure is not a screenshot of the steps that caused it.
Small or low-contrast text. Fine print, dense code at low resolution, and light grey placeholder text are where misreads cluster.
Which part you care about. On a full-page screenshot, the model has no idea the problem is the misaligned button in the corner.
Real values behind a rendering. A date shown as 03/04/2026 is genuinely ambiguous, and the model will pick a convention.
Providers document the practical limits, including resolution handling and how images are counted for cost. OpenAI's images and vision guide is worth ten minutes if you are sending screenshots at volume, because image tokens add up faster than most people expect.
The four-part caption
Every screenshot prompt that works has these four things. Missing any one of them is where the attempts get wasted.
What the image is. "This is the checkout page of a React app on mobile Safari."
Where to look. "Look at the total row under the item list."
What is wrong or wanted. "The total shows 0.00 but the items add up to 48.50."
What you want back. "Tell me the three most likely causes in the component tree, do not write code yet."
Compare that to what people usually send, which is the screenshot plus "why is this broken". The model then has to guess the platform, guess the region of interest, guess the intended behaviour, and guess whether you want a diagnosis or a patch. It will guess plausibly and confidently, and you will spend two more messages correcting each guess.
Worked example: a layout bug
A concrete before and after, using the same image.
Weak:
[screenshot] this looks wrong, fix it
Strong:
[screenshot]
Desktop Chrome at 1440px, our settings page.
The right column card overlaps the footer at the bottom of the page.
Expected: the footer sits below all content with 48px of clearance.
The layout uses CSS grid, footer is outside the grid container.
Give me the two most likely causes. Do not rewrite the component.The strong version costs fifteen seconds to type and typically saves two rounds. The instruction not to write code yet matters more than it looks: without it, models tend to jump to a full rewrite of the component, which is both harder to review and often unnecessary.
Cropping beats explaining
If you find yourself writing a long sentence to describe where to look, crop instead. A tight crop around the problem area outperforms a full-page screenshot plus directions, for a simple reason: the model is not searching for the subject any more.
Two exceptions where the full view wins. Layout and alignment problems need surrounding context, because the bug is the relationship between elements. And anything where the answer might be "the problem is elsewhere on the page" needs the page.
When you genuinely need both, send both: the full view for context and the crop for focus, and say which is which.
Screenshots for building, not just debugging
Handing a design screenshot to a coding agent and asking for the component works, with one adjustment. Say what should be flexible. A model reproducing a screenshot will otherwise reproduce it literally, hard-coding the exact strings, the exact number of rows, and the specific widths it can see.
Useful phrasing: "Build this as a component. The three cards are sample data from a prop, not fixed content. Match spacing and hierarchy, use our existing button component rather than recreating the button in the image."
The same discipline applies when the screenshot is evidence for a coding agent rather than a design. How to write a bug report for an AI coding agent covers what to include alongside the image, and if the agent needs to know how your product works generally rather than what one screen looks like, that is a context problem, covered in how to give AI context about your business.
A note on what you are pasting
Screenshots leak more than text does. A screenshot of your admin panel contains real customer names, real order values, and often a URL with an internal hostname in it. A screenshot of your IDE contains whatever else was open, including the file with the credentials in it.
Crop before sending, not after. It is the same habit as redacting a log, and it fails the same way when you rely on remembering to do it. If the tool is a general assistant rather than something inside your own infrastructure, assume the image is retained under whatever policy the vendor publishes.
The rest of the discipline behind good prompting is in prompt engineering, which applies unchanged when the input happens to be an image.
Frequently asked questions
Is a screenshot better than pasting the code?
For visual problems, yes. For logic problems, no. Paste code when the question is "why does this function return the wrong value" and send a screenshot when the question is "why does this look wrong". Send both when the answer depends on the link between them.
What resolution should I send?
Enough that text is comfortably legible when you view it at 100 percent. Beyond that you are paying for tokens without adding information. Very large images get downscaled by the provider anyway, which can push small text below the readable threshold.
Can AI read a photo of a screen taken with my phone?
Usually, if it is in focus and roughly square on. Glare, keystone distortion, and moire patterns cause misreads, and misreads on a screenshot are worse than failures because the model proceeds confidently from the wrong text.
Should I annotate the screenshot with arrows?
It helps, and models follow drawn circles and arrows well. A crop usually achieves the same thing with less effort, but annotation wins when you need to reference three separate spots in one image.
How did this land?
About the author
Developer Advocate
Steve builds something with Swarmz every week and writes up what worked, what broke, and what he'd do differently. Tutorials and hands-on guides are his lane.
