UpfynUpfyn
Upfyn Web

Recipes

Handle errors and retries

A wrapper you can put in front of every call. The value is not in the retrying — it is in knowing what not to retry.

The wrapper#

resilient.py
import random
import time
from openai import OpenAI, APIStatusError, APIConnectionError, APITimeoutError
 
client = OpenAI(
    base_url="https://ai.upfyn.com/v1",
    api_key=KEY,
    timeout=60.0,
    max_retries=0,        # we do this ourselves; the SDK would retry a 402
)
 
# Retrying anything else is either pointless or harmful.
RETRYABLE_STATUS = {429, 500, 502, 503, 504}
 
 
class OutOfCredits(Exception):
    """402. No amount of retrying fixes this."""
 
 
class BadRequest(Exception):
    """400/403/404. Your request is wrong; it will be wrong next time too."""
 
 
def complete(*, attempts: int = 5, **kwargs):
    for attempt in range(attempts):
        try:
            return client.chat.completions.create(**kwargs)
 
        except APIStatusError as err:
            status = err.status_code
 
            if status == 402:
                raise OutOfCredits(_message(err)) from err
            if status in (400, 401, 403, 404, 413):
                raise BadRequest(f"{status}: {_message(err)}") from err
            if status not in RETRYABLE_STATUS or attempt == attempts - 1:
                raise
 
            time.sleep(_backoff(err, attempt))
 
        except (APIConnectionError, APITimeoutError):
            # The request may never have reached us. Safe to retry.
            if attempt == attempts - 1:
                raise
            time.sleep(_backoff(None, attempt))
 
    raise RuntimeError("unreachable")
 
 
def _backoff(err, attempt: int) -> float:
    """Honour Retry-After when the server sent one; otherwise exponential with jitter."""
    if err is not None:
        header = err.response.headers.get("Retry-After")
        if header:
            try:
                return min(float(header), 60.0)
            except ValueError:
                pass
    # Jitter matters: without it, every client that failed together retries together.
    return min(2 ** attempt, 30) + random.random()
 
 
def _message(err) -> str:
    try:
        return err.response.json()["error"]["message"]
    except Exception:
        return str(err)

What to retry, and what never to#

StatusRetry?Why
429YesRate limited. Honour Retry-After; retrying harder makes it worse.
500 / 502 / 503 / 504YesOurs. Back off and try again.
Connection / timeoutYesIt may never have reached us.
400NoYour request is malformed. It will be malformed next time.
401NoBad credentials. A retry sends the same bad credentials.
402NoOut of credits. Retrying just burns time and logs noise.
403NoNot permitted. Retrying will not grant permission.
413NoThe file is too big and will stay too big.

Turn the SDK's retries off

Both OpenAI SDKs retry by default, and their policy does not know that a 402 is hopeless. Set max_retries=0 and own the decision, or you will silently make three failing calls where one would do.

Backoff with jitter#

The jitter is not decoration. When a burst of requests hits a rate limit together, they all fail together — and without jitter they all retry at the same instant and fail together again. A random fraction of a second spreads them out.

  • Prefer Retry-After when it is present. The server knows better than your formula.
  • Cap the wait. An unbounded exponential eventually sleeps for hours.
  • Cap the attempts. Five is plenty; if five fail, something is actually wrong.

Not paying twice#

If a response is lost in transit, your retry could be billed as a second request. Idempotency-Key prevents that — reuse the same key for every attempt at one logical request.

Idempotency-Key
import uuid
 
# Same key on every attempt of the SAME logical request. A repeat is not billed twice.
key = str(uuid.uuid4())
 
complete(
    model="upfyn-yuva",
    messages=messages,
    extra_headers={"Idempotency-Key": key},
)

Failing before you start#

For expensive work, check the balance first. Discovering you are empty halfway through a batch is worse than not starting it.

A pre-flight check
def complete_with_budget(budget_credits: float, **kwargs):
    """Refuse to start work you cannot afford to finish."""
    me = client.get("/me", cast_to=dict)
    if me["total_credits"] < budget_credits:
        raise OutOfCredits(f"need {budget_credits}, have {me['total_credits']}")
    return complete(**kwargs)

The full list#

Every status, type and code the API can return is on Errors, including what a mid-stream failure looks like — which is the case this wrapper does not cover, because by then the response is already a 200.

PreviousRead a screenshot