Read a screenshot

Sending an image and asking a question about it — a stack trace, an invoice, a chart, a design comparison.

The call#

vision.py

1import base64
2from openai import OpenAI
3 
4client = OpenAI(base_url="https://ai.upfyn.com/v1", api_key=KEY, max_retries=0)
5 
6 
7def read_screenshot(path: str, question: str) -> str:
8    with open(path, "rb") as f:
9        b64 = base64.b64encode(f.read()).decode()
10 
11    response = client.chat.completions.create(
12        model="upfyn-yuva",              # Bala is text-only and would 400 here
13        messages=[{
14            "role": "user",
15            "content": [
16                {"type": "text", "text": question},
17                {"type": "image_url",
18                 "image_url": {"url": f"data:image/png;base64,{b64}"}},
19            ],
20        }],
21        max_tokens=600,
22        temperature=0,
23    )
24    return response.choices[0].message.content
25 
26 
27print(read_screenshot(
28    "error.png",
29    "What is the exact error message, and which line of the stack trace is in our own code?",
30))

“Describe this image” produces a paragraph you have to read. “What is the exact error message, and which line is in our own code?” produces the two facts you wanted. The question does more for accuracy than the model choice does.

Do not shrink it

Downscaling is the most common cause of a wrong answer

Compressing a dense screenshot until the text blurs means the model is guessing at characters. If you are reading small text, send it at full size — the extra tokens cost less than a wrong answer.

Ask for fields, not prose

If you are going to parse the answer, get it structured in the first place.

Vision plus a schema

1SCHEMA = {
2    "type": "object",
3    "properties": {
4        "error_message": {"type": "string"},
5        "error_type":    {"type": "string"},
6        "file":          {"type": "string"},
7        "line":          {"type": ["integer", "null"]},
8        "is_our_code":   {"type": "boolean"},
9    },
10    "required": ["error_message", "error_type", "file", "line", "is_our_code"],
11    "additionalProperties": False,
12}
13 
14response = client.chat.completions.create(
15    model="upfyn-yuva",
16    messages=[{
17        "role": "user",
18        "content": [
19            {"type": "text", "text": "Read this stack trace."},
20            {"type": "image_url", "image_url": {"url": data_uri}},
21        ],
22    }],
23    response_format={"type": "json_schema",
24                     "json_schema": {"name": "trace", "schema": SCHEMA, "strict": True}},
25)

Note "line": {"type": ["integer", "null"]} — sometimes there is no line number, and a nullable field is honest where a required integer forces the model to invent one.

Several images at once

Comparing two images

1content = [
2    {"type": "text",
3     "text": "The first image is the design, the second is what shipped. "
4             "List every visual difference."},
5    {"type": "image_url", "image_url": {"url": design_uri}},
6    {"type": "image_url", "image_url": {"url": built_uri}},
7]

Say in the text which image is which. Order alone is not a reliable label.

Which model#

Yuva for almost everything — images, PDFs, documents.
Rishi when the reasoning about the image is hard, or the input is video.
Bala cannot see. Attaching an image returns 400 media_not_supported rather than being ignored.

Reusing an image#

Base64 counts toward every request that carries it. If you will ask several questions about the same file, upload it once and reference the file_id — see Multimodal input.