Talking to an LLM API From Production Code Without the Sharp Edges
A dependency-light LLM client for a cron job: real headers, explicit timeouts, JSON mode with shape validation, a correction loop, and a fake provider to prove it works.
A nightly pipeline of ours started failing with HTTP 403 error code 1010 from every provider tier at once. Every tier failing together reads as an account problem, so we went and checked keys and credits. It was neither: the client posted with urllib.request and no User-Agent, so the request went out as Python-urllib/3.10 and the CDN in front of the provider dropped it. curl worked.
The diagnostic order
Curl the endpoint first. If curl works and your code does not, suspect your headers before your keys, your credits or a block.
Be honest, too, about what a tier chain buys. Two keys on one endpoint survive a per-key rate limit. They did not survive that CDN block: they shared the broken thing. Only a different provider survives an outage.
The request
An OpenAI-compatible /v1/chat/completions POST needs three things first drafts leave out. A real user agent. An explicit timeout on every call, because urlopen has no default one and a nightly job that hangs stays hung until morning. And the key read from the environment, never committed and never logged. Reading the response has its own trap: handle HTTPError separately from URLError and call e.read(), because the provider's real error message is in the error body.
"""A small LLM HTTP client for a cron job. Standard library only."""
import json
import os
import random
import re
import time
import urllib.error
import urllib.request
USER_AGENT = 'agicad-pipeline/1.0 (+https://example.com/)'
RETRYABLE = frozenset({408, 409, 425, 429, 500, 502, 503, 504})
TERMINAL = frozenset({400, 401, 402, 403, 404, 422})
FENCE_RE = re.compile(r'^\s*```(?:json)?\s*(.*?)\s*```\s*$', re.S)
class Tier:
"""One endpoint the pipeline is allowed to use."""
def __init__(self, name, base_url, model, key_env):
self.name, self.model, self.key_env = name, model, key_env
self.base_url = base_url.rstrip('/')
@property
def key(self):
return os.environ.get(self.key_env, '')
class TierError(RuntimeError):
def __init__(self, tier, status, message, retryable):
super().__init__('%s: %s %s' % (tier.name, status, message))
self.tier, self.status, self.retryable = tier, status, retryable
class ValidationError(ValueError):
"""The model returned something, but not the document we asked for."""
SECRET_HEADERS = ('authorization', 'x-api-key', 'api-key')
def redact(headers):
"""Headers safe to log. Never print an Authorization header."""
return {k: (v[:7] + '...redacted' if v else '...empty')
if k.lower() in SECRET_HEADERS else v for k, v in headers.items()}
def strip_fence(text):
"""Remove a ```json ... ``` wrapper. Models add it even in JSON mode."""
match = FENCE_RE.match(text or '')
return match.group(1) if match else (text or '').strip()
JSON mode is not a guarantee
Providers that accept response_format: {"type": "json_object"} hand back a string you still have to parse. Accepting the parameter is not a promise the string is valid JSON, and certainly not that the document has the shape you wanted. Two surprises recur: a stray ```json fence, and a valid document with the wrong types in it.
So: strip the fence, json.loads, then validate the shape yourself - required keys, types, element types, length bounds. The messages are written for a model to read, which is the point.
def check_shape(doc, spec):
"""Validate a decoded document: spec maps a key to (type, required)
or (list, element_type, min, max). The message is fed back to the model."""
if not isinstance(doc, dict):
raise ValidationError('top level must be a JSON object, got %s'
% type(doc).__name__)
problems = []
for key, rule in spec.items():
if key not in doc:
if rule[-1] is True or rule[0] is list:
problems.append('missing required key %r' % key)
continue
value = doc[key]
if rule[0] is list:
_, element_type, low, high = rule
if not isinstance(value, list):
problems.append('%r must be a list, got %s'
% (key, type(value).__name__))
continue
if not low <= len(value) <= high:
problems.append('%r must have %d to %d items, got %d'
% (key, low, high, len(value)))
for i, item in enumerate(value):
if not isinstance(item, element_type):
problems.append('%s[%d] must be %s, got %s'
% (key, i, element_type.__name__,
type(item).__name__))
else:
wanted = rule[0]
if not isinstance(value, wanted) or isinstance(value, bool) != (wanted is bool):
problems.append('%r must be %s, got %s'
% (key, wanted.__name__, type(value).__name__))
extra = sorted(k for k in doc if k not in spec)
if extra:
problems.append('unexpected key(s): %s' % ', '.join(extra))
if problems:
raise ValidationError('; '.join(problems))
return doc
On a validation failure you feed the failure back as a correction turn and ask again, up to N attempts, then fail loudly. Here is the client, with the tier chain, the retry rules and that loop.
class LLMClient:
def __init__(self, tiers, timeout=45.0, max_shape_retries=2,
max_http_attempts=3, temperature=0.0, sleep=time.sleep,
rng=random, logger=print):
self.tiers, self.timeout, self.temperature = list(tiers), timeout, temperature
self.max_shape_retries, self.max_http_attempts = max_shape_retries, max_http_attempts
self.sleep, self.rng, self.log, self.calls = sleep, rng, logger, 0
# -- transport ------------------------------------------------------
def _post(self, tier, payload):
body = json.dumps(payload).encode('utf-8')
headers = {
'Content-Type': 'application/json',
# Without this a CDN in front of the provider sees
# Python-urllib/3.x and may block the request outright.
'User-Agent': USER_AGENT,
'Authorization': 'Bearer %s' % tier.key,
}
self.log(' -> POST %s (%s)' % (tier.base_url,
redact(headers)['Authorization']))
request = urllib.request.Request(
tier.base_url + '/v1/chat/completions', data=body,
headers=headers, method='POST')
self.calls += 1
try:
# An explicit timeout on EVERY call. The default is no timeout,
# which is how a nightly job hangs until somebody notices.
with urllib.request.urlopen(request, timeout=self.timeout) as response:
return json.loads(response.read().decode('utf-8'))
except urllib.error.HTTPError as exc:
detail = exc.read().decode('utf-8', 'replace')[:300]
raw = exc.headers.get('Retry-After') or ''
error = TierError(tier, exc.code, detail,
exc.code in RETRYABLE and exc.code not in TERMINAL)
error.retry_after = float(raw) if raw.strip().isdigit() else None
raise error
except urllib.error.URLError as exc:
raise TierError(tier, 'conn', repr(exc.reason), True)
except TimeoutError as exc:
raise TierError(tier, 'timeout', repr(exc), True)
def _complete(self, tier, messages):
payload = {
'model': tier.model,
'messages': messages,
'temperature': self.temperature,
'response_format': {'type': 'json_object'},
}
for attempt in range(self.max_http_attempts):
try:
data = self._post(tier, payload)
except TierError as exc:
if not exc.retryable:
self.log(' !! %s (terminal, moving to the next tier)' % exc)
raise
wait = getattr(exc, 'retry_after', None)
if wait is None:
wait = self.rng.uniform(0.0, min(20.0, 0.5 * 2 ** attempt))
self.log(' !! %s (retryable, sleeping %.2fs)' % (exc, wait))
if attempt + 1 >= self.max_http_attempts:
raise
self.sleep(wait)
continue
return data['choices'][0]['message']['content']
raise TierError(tier, 'exhausted', 'no attempts left', False)
# -- the public call ------------------------------------------------
def structured(self, system, user, spec):
"""A validated dict, or an exception once every tier is exhausted."""
last_error = None
for tier in self.tiers:
if not tier.key:
self.log('== tier %s: no key in %s, skipping'
% (tier.name, tier.key_env))
continue
self.log('== tier %s (%s)' % (tier.name, tier.model))
messages = [{'role': 'system', 'content': system},
{'role': 'user', 'content': user}]
try:
for round_number in range(self.max_shape_retries + 1):
raw = self._complete(tier, messages)
try:
doc = json.loads(strip_fence(raw))
except ValueError as exc:
problem = 'the response was not valid JSON: %s' % exc
else:
try:
return check_shape(doc, spec)
except ValidationError as exc:
problem = str(exc)
self.log(' ?? invalid (round %d): %s' % (round_number + 1, problem))
last_error = ValidationError(problem)
if round_number >= self.max_shape_retries:
break
# Feed the validation error back as a correction turn.
messages = messages + [
{'role': 'assistant', 'content': raw},
{'role': 'user', 'content': 'Rejected: %s. Reply with the '
'corrected JSON only.' % problem},
]
except TierError as exc:
last_error = exc
continue
raise RuntimeError('every tier failed; last error: %r' % (last_error,))
The retry rules matter, because getting them backwards burns money. 429, 5xx and timeouts are retryable on the same tier. 401, 402 and 400 are terminal - a bad key does not become a good key on retry - so drop that tier and try the next.
Proving it without a provider
You cannot make a real provider return a 429 on demand. So we run this against a fake endpoint that misbehaves to order.
"""A fake OpenAI-compatible endpoint that misbehaves to order.
Each POST to /v1/chat/completions returns the next item of SCRIPT; the last
entry repeats. Pass your own script to choose:
python fake_provider.py 8098 wrong-shape,good
"""
import json
import sys
from http.server import BaseHTTPRequestHandler, HTTPServer
GOOD = {'title': 'Cheaper Nightly Builds', 'tags': ['ci', 'cost'], 'score': 7}
SCRIPT = ['good', 'fenced', 'wrong-shape', '429', '500', '401']
def completion(content):
return {'id': 'chatcmpl-fake', 'object': 'chat.completion',
'choices': [{'index': 0, 'finish_reason': 'stop',
'message': {'role': 'assistant', 'content': content}}]}
class Handler(BaseHTTPRequestHandler):
step = 0
def log_message(self, *args):
pass
def _send(self, code, payload, headers=()):
body = json.dumps(payload).encode('utf-8')
self.send_response(code)
self.send_header('Content-Type', 'application/json')
for key, value in headers:
self.send_header(key, value)
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_POST(self):
length = int(self.headers.get('Content-Length') or 0)
request = json.loads(self.rfile.read(length) or b'{}')
step = SCRIPT[min(Handler.step, len(SCRIPT) - 1)]
Handler.step += 1
print(' provider: call %d, model=%s, %d message(s) -> %s'
% (Handler.step, request.get('model'),
len(request.get('messages', [])), step), flush=True)
if step == 'good':
return self._send(200, completion(json.dumps(GOOD)))
if step == 'fenced':
return self._send(200, completion('```json\n%s\n```' % json.dumps(GOOD)))
if step == 'wrong-shape':
return self._send(200, completion(json.dumps(
{'title': 123, 'tags': 'ci, cost', 'notes': 'extra'})))
if step == '429':
return self._send(429, {'error': {'message': 'rate limit'}},
[('Retry-After', '1')])
if step == '500':
return self._send(500, {'error': {'message': 'upstream exploded'}})
return self._send(401, {'error': {'message': 'invalid api key'}})
if __name__ == '__main__':
port = int(sys.argv[1]) if len(sys.argv) > 1 else 8098
if len(sys.argv) > 2:
SCRIPT[:] = sys.argv[2].split(',')
HTTPServer(('127.0.0.1', port), Handler).serve_forever()
The unit-level pieces first - fence stripping and the shape check:
$ venv/bin/python drive.py
== strip_fence ==
'{"a": 1}' -> '{"a": 1}'
'```json\n{"a": 1}\n```' -> '{"a": 1}'
'```\n{"a": 1}\n```' -> '{"a": 1}'
== check_shape ==
PASS {'title': 'ok', 'tags': ['a'], 'score': 3}
FAIL {'title': 123, 'tags': 'a, b', 'notes': 'x'} -> 'title' must be str, got int; 'tags' must be a list, got str; missing required key 'score'; unexpected key(s): notes
FAIL {'title': 'ok', 'tags': ['a', 'b'], 'score': True} -> 'score' must be int, got bool
Then the client, against instances running different scripts. Real output:
$ venv/bin/python drive.py
--- 3. wrong shape -> correction round -> 429 -> 500 ---
== tier primary (fake-large)
-> POST http://127.0.0.1:8098 (Bearer ...redacted)
?? invalid (round 1): 'title' must be str, got int; 'tags' must be a list, got str; missing required key 'score'; unexpected key(s): notes
-> POST http://127.0.0.1:8098 (Bearer ...redacted)
!! primary: 429 {"error": {"message": "rate limit"}} (retryable, sleeping 1.00s)
-> POST http://127.0.0.1:8098 (Bearer ...redacted)
!! primary: 500 {"error": {"message": "upstream exploded"}} (retryable, sleeping 0.24s)
RAISED: RuntimeError every tier failed; last error: TierError('primary: 500 {"error": {"message": "upstream exploded"}}')
http calls made: 3
--- 5. 401 is terminal: no retry, straight to the next tier, which then self-corrects ---
== tier dead-key (fake-large)
-> POST http://127.0.0.1:8095 (Bearer ...redacted)
!! dead-key: 401 {"error": {"message": "invalid api key"}} (terminal, moving to the next tier)
== tier recovering (fake-small)
-> POST http://127.0.0.1:8094 (Bearer ...redacted)
?? invalid (round 1): 'title' must be str, got int; 'tags' must be a list, got str; missing required key 'score'; unexpected key(s): notes
-> POST http://127.0.0.1:8094 (Bearer ...redacted)
RESULT: {'title': 'Cheaper Nightly Builds', 'tags': ['ci', 'cost'], 'score': 7}
http calls made: 3
slept for: [1.0, 1.0]
Run 3: a correction round, a 429 whose Retry-After is obeyed to the second, a 500, a loud failure. Run 5: a 401 dropping its tier without retrying, and the next tier corrected into a valid document.
Operational hazards
Non-determinism is the one that bites. Set temperature explicitly rather than inheriting a provider default that can change. Log the exact request and response pair for anything written to a database. And treat model output as untrusted input: never eval it, never interpolate it into SQL or a shell command, escape it before it reaches HTML.
Then budget: count attempts, cap them, and make the failure mode "skip this item and keep going" rather than "abort the batch". And because this site carries an EU AI Act transparency disclosure: anything published from a pipeline like this should be disclosed as AI-assisted where the law or your own policy requires it.
When not to do this
If a deterministic parser, a regex or a lookup table solves the problem, use it. An LLM is the most expensive and least predictable way to do something a str.split() does perfectly, every time, for free. The test is whether the input is genuinely open-ended: classifying free-text feedback, yes; pulling a date out of a field that is always an ISO date, no.
If you have an LLM feature that works in a notebook and falls over on a schedule, or a content pipeline that must run unattended, that is the gap we close. Start a project.
We ship and run our own products. We build yours the same way.
Start a project