A Scraper That Is Polite, Cheap, and Still Works Tomorrow
Conditional requests, full-jitter backoff, a per-host throttle and a circuit breaker, all in the standard library, tested against a local server that misbehaves on purpose.
A Python script of ours started getting 403s from a host behind Cloudflare. Error 1010. The same URL in curl returned 200 every time. It looked exactly like a credentials problem, and it was not: the script sent no User-Agent, so the request went out as Python-urllib/3.10, and the edge blocked it on sight.
The diagnostic order that matters
That bug was worth the hour, because it fixes a diagnostic order. When your code gets a 4xx and you are about to check your API key, your billing page or whether you have been blocked: curl the endpoint first. If curl works and your code does not, the difference is in your headers, not your account. Two minutes of that saves an afternoon of looking at the wrong thing.
Identify yourself, and read robots.txt
A real User-Agent naming the bot and a contact URL costs nothing and buys the chance of an email instead of a block. urllib.robotparser handles both can_fetch and Crawl-delay, and we take the crawl delay as the host's own answer to "how fast may a bot go".
Be honest about what that buys: robots.txt is a convention, not a permission system. Whether you are allowed to fetch a site is a question about its terms of service and your jurisdiction - the owner's call, not your HTTP library's. Obeying robots.txt is the technical minimum, not the legal answer.
Conditional requests are the big win
On a recurring crawl the thing that actually costs you - bandwidth, time, goodwill - is re-fetching pages that have not changed. On CijeneHR, one of our products and a price-comparison site, most pages on most days are byte-identical to yesterday's.
So store the ETag and Last-Modified per URL, send them back as If-None-Match and If-Modified-Since, and treat a 304 as a cheap success rather than an error. You still need the previous body, so cache that on disk too - which also means re-running during development hammers nobody.
Backoff with full jitter
Retry only on 429, 5xx and connection errors. A 404 is an answer; a 400 is your bug. Retrying either wastes your budget and looks like an attack.
When you do retry, use exponential backoff with full jitter: sleep = random.uniform(0, min(cap, base * 2 ** n)). The whole interval is random, not an exponential value with noise sprinkled on. A batch of workers usually fails together, and if they all back off by the same computed amount they come back together and recreate the spike that knocked the host over.
And when the server tells you how long to wait, listen. Retry-After comes in two forms - a delay in seconds, or an HTTP-date - and code that handles only the first silently ignores the polite half of the internet.
"""A polite, cheap, resilient HTTP fetcher. Standard library only."""
import email.utils
import hashlib
import json
import os
import random
import time
import urllib.error
import urllib.parse
import urllib.request
import urllib.robotparser
USER_AGENT = 'ExampleBot/1.0 (+https://example.com/bot)'
RETRY_STATUSES = frozenset({408, 425, 429, 500, 502, 503, 504})
class Fetched:
def __init__(self, url, status, body, from_cache=False, attempts=1):
self.url, self.status, self.body = url, status, body
self.from_cache, self.attempts = from_cache, attempts
def __repr__(self):
return '<Fetched %s %s %s bytes%s>' % (
self.status, self.url, len(self.body or b''),
' (cache)' if self.from_cache else '')
class CircuitOpen(RuntimeError):
"""The host is in cooldown; we are not calling it."""
def parse_retry_after(value, now=None):
"""Seconds to wait, from either Retry-After form: a delay or an HTTP-date."""
if not value:
return None
value = value.strip()
if value.isdigit():
return float(value)
try:
when = email.utils.parsedate_to_datetime(value)
except (TypeError, ValueError):
return None
if when is None:
return None
if when.tzinfo is None:
import datetime
when = when.replace(tzinfo=datetime.timezone.utc)
now = time.time() if now is None else now
return max(0.0, when.timestamp() - now)
def backoff_delay(attempt, base=0.5, cap=30.0, rng=random):
"""Exponential backoff with FULL jitter: the whole interval is random."""
return rng.uniform(0.0, min(cap, base * (2 ** attempt)))
That parser, against both forms and some garbage:
== Retry-After parser ==
'120' -> 120.0
' 30 ' -> 30.0
'Wed, 21 Oct 2026 07:28:00 GMT' -> 480.0
'Wed, 21 Oct 2026 07:00:00 GMT' -> 0.0
'soon' -> None
'' -> None
None -> NoneAnd the schedule it generates. Note the sampled column: attempt 2 slept 15 milliseconds, attempt 6 twenty seconds. That spread is the point.
== backoffschedule, full jitter, base=0.5 cap=30 ==
attempt ceiling sampled
0 0.50 0.483
1 1.00 0.441
2 2.00 0.015
3 4.00 3.644
4 8.00 7.514
5 16.00 9.316
6 30.00 20.147
7 30.00 2.518The rest of the client
Two more pieces round it out. A per-host minimum interval, seeded from Crawl-delay when the site publishes one, so one slow host is not hammered by a concurrent run. And a circuit breaker: after N consecutive failures for a host, stop calling it for a cooldown instead of spending the whole run's budget on one broken domain while forty other sites go unfetched.
class Store:
"""Per-URL validators and last known body, on disk."""
def __init__(self, directory):
self.directory = directory
os.makedirs(os.path.join(directory, 'bodies'), exist_ok=True)
self.path = os.path.join(directory, 'validators.json')
self.data = {}
if os.path.exists(self.path):
try:
with open(self.path, encoding='utf-8') as fh:
self.data = json.load(fh)
except ValueError:
pass
def conditional_headers(self, url):
entry = self.data.get(url) or {}
headers = {}
if entry.get('etag'):
headers['If-None-Match'] = entry['etag']
if entry.get('last_modified'):
headers['If-Modified-Since'] = entry['last_modified']
return headers
def remember(self, url, response_headers, body):
entry = self.data.setdefault(url, {})
for name, key in (('ETag', 'etag'), ('Last-Modified', 'last_modified')):
if response_headers.get(name):
entry[key] = response_headers[name]
tmp = self.path + '.tmp'
with open(tmp, 'w', encoding='utf-8') as fh:
json.dump(self.data, fh, indent=1, sort_keys=True)
os.replace(tmp, self.path)
with open(self._body_path(url), 'wb') as fh:
fh.write(body)
def _body_path(self, url):
digest = hashlib.sha256(url.encode('utf-8')).hexdigest()[:32]
return os.path.join(self.directory, 'bodies', digest + '.bin')
def cached_body(self, url):
path = self._body_path(url)
if not os.path.exists(path):
return None
with open(path, 'rb') as fh:
return fh.read()
class HostState:
"""Minimum interval plus circuit breaker for one host."""
def __init__(self, min_interval, failure_threshold, cooldown):
self.min_interval = min_interval
self.failure_threshold = failure_threshold
self.cooldown = cooldown
self.last_request = 0.0
self.failures = 0
self.open_until = 0.0
def before_request(self, sleep, now):
if now() < self.open_until:
raise CircuitOpen('circuit open for another %.1fs'
% (self.open_until - now()))
wait = self.min_interval - (now() - self.last_request)
if wait > 0:
sleep(wait)
def record(self, ok, now):
self.last_request = now()
if ok:
self.failures = 0
return
self.failures += 1
if self.failures >= self.failure_threshold:
self.open_until = now() + self.cooldown
class PoliteFetcher:
def __init__(self, cache_dir, user_agent=USER_AGENT, min_interval=1.0,
max_attempts=4, failure_threshold=3, cooldown=60.0,
timeout=20.0, obey_robots=True, sleep=time.sleep,
now=time.time, rng=random):
self.user_agent, self.timeout = user_agent, timeout
self.default_min_interval = min_interval
self.max_attempts = max_attempts
self.failure_threshold, self.cooldown = failure_threshold, cooldown
self.obey_robots = obey_robots
self.sleep, self.now, self.rng = sleep, now, rng
self.store = Store(cache_dir)
self.hosts, self.robots, self.log = {}, {}, []
def _robots(self, url):
parts = urllib.parse.urlsplit(url)
origin = '%s://%s' % (parts.scheme, parts.netloc)
if origin not in self.robots:
parser = urllib.robotparser.RobotFileParser()
parser.set_url(origin + '/robots.txt')
try:
parser.read()
except Exception:
parser.parse([]) # missing robots.txt conventionally means allowed
self.robots[origin] = parser
return self.robots[origin]
def allowed(self, url):
return True if not self.obey_robots else \
self._robots(url).can_fetch(self.user_agent, url)
def crawl_delay(self, url):
if not self.obey_robots:
return None
value = self._robots(url).crawl_delay(self.user_agent)
return float(value) if value is not None else None
def _host_state(self, url):
host = urllib.parse.urlsplit(url).netloc
if host not in self.hosts:
interval = self.crawl_delay(url) or self.default_min_interval
self.hosts[host] = HostState(interval, self.failure_threshold,
self.cooldown)
return self.hosts[host]
def _open(self, url):
headers = {'User-Agent': self.user_agent, 'Accept-Encoding': 'identity',
'Accept': 'text/html,application/xhtml+xml,*/*;q=0.8'}
headers.update(self.store.conditional_headers(url))
request = urllib.request.Request(url, headers=headers)
return urllib.request.urlopen(request, timeout=self.timeout)
def fetch(self, url):
if not self.allowed(url):
self.log.append((url, 'robots-disallow', None))
return Fetched(url, 0, None)
state, last_error = self._host_state(url), None
for attempt in range(self.max_attempts):
try:
state.before_request(self.sleep, self.now)
except CircuitOpen as exc:
self.log.append((url, 'circuit-open', str(exc)))
cached = self.store.cached_body(url)
return Fetched(url, 0, cached, cached is not None, attempt)
try:
with self._open(url) as response:
body = response.read()
self.store.remember(url, response.headers, body)
state.record(True, self.now)
self.log.append((url, response.status, 'ok'))
return Fetched(url, response.status, body, False, attempt + 1)
except urllib.error.HTTPError as exc:
body = exc.read()
if exc.code == 304:
state.record(True, self.now)
self.log.append((url, 304, 'not-modified'))
return Fetched(url, 304, self.store.cached_body(url),
True, attempt + 1)
if exc.code not in RETRY_STATUSES:
state.record(True, self.now) # a 404 is an answer, not a fault
self.log.append((url, exc.code, 'terminal'))
return Fetched(url, exc.code, body, False, attempt + 1)
state.record(False, self.now)
last_error = exc
wait = parse_retry_after(exc.headers.get('Retry-After'), self.now())
if wait is None:
wait = backoff_delay(attempt, rng=self.rng)
self.log.append((url, exc.code, 'retry in %.2fs' % wait))
except (urllib.error.URLError, OSError) as exc:
state.record(False, self.now)
last_error = exc
wait = backoff_delay(attempt, rng=self.rng)
self.log.append((url, 'conn-error', 'retry in %.2fs' % wait))
if attempt + 1 < self.max_attempts:
self.sleep(wait)
self.log.append((url, 'gave-up', repr(last_error)))
cached = self.store.cached_body(url)
return Fetched(url, 0, cached, cached is not None, self.max_attempts)Verify against something that misbehaves
A fixture recorded from a broken run encodes the breakage, and you then spend a day making your code pass a test that asserts the bug. Verify against a live endpoint before trusting any fixture - and for the failure paths, stand up a local server that produces them on demand. This is fifty lines and it is the most valuable file in the project.
"""A local site that misbehaves on purpose, so the client can be tested.
python fake_site.py 8099
"""
import sys
from http.server import BaseHTTPRequestHandler, HTTPServer
ETAG = '"v1-abc"'
BODY = b'<html><body><p>price: 19.99</p></body></html>'
SCRIPT = ['200', '304', '429', '500', '200']
class Handler(BaseHTTPRequestHandler):
step = 0
def log_message(self, *args):
pass
def _send(self, code, headers=(), body=b''):
self.send_response(code)
for key, value in headers:
self.send_header(key, value)
self.send_header('Content-Length', str(len(body)))
self.end_headers()
if body:
self.wfile.write(body)
def do_GET(self):
if self.path == '/robots.txt':
return self._send(200, [('Content-Type', 'text/plain')],
b'User-agent: *\nCrawl-delay: 1\nDisallow: /private\n')
if self.path == '/gone':
return self._send(404, [('Content-Type', 'text/plain')], b'no such page')
if self.path == '/private':
return self._send(200, [], b'should never be fetched')
if self.path != '/page':
return self._send(404, [], b'')
step = SCRIPT[min(Handler.step, len(SCRIPT) - 1)]
Handler.step += 1
ua = self.headers.get('User-Agent', '')
print(' server: request %d, UA=%r, If-None-Match=%r -> %s'
% (Handler.step, ua, self.headers.get('If-None-Match'), step),
flush=True)
if step == '200':
return self._send(200, [('Content-Type', 'text/html'), ('ETag', ETAG),
('Last-Modified', 'Wed, 21 Oct 2026 07:28:00 GMT')],
BODY)
if step == '304':
if self.headers.get('If-None-Match') == ETAG:
return self._send(304, [('ETag', ETAG)], b'')
return self._send(200, [('ETag', ETAG)], BODY)
if step == '429':
return self._send(429, [('Retry-After', '2')], b'slow down')
return self._send(500, [], b'boom')
if __name__ == '__main__':
port = int(sys.argv[1]) if len(sys.argv) > 1 else 8099
HTTPServer(('127.0.0.1', port), Handler).serve_forever()Pointing the client at it gives a transcript you can read: a 200 that stores the validator, a 304 that costs nothing, a 404 that is not retried, a 429 whose Retry-After is obeyed rather than guessed at, a 500 that gets a jittered retry, and then the breaker refusing to call the host at all.
== live client against the local fake site ==
robots allows /page : True
robots allows /private: False
crawl delay : 1.0
first fetch -> <Fetched 200 http://127.0.0.1:8099/page 45 bytes> attempts=1
re-fetch (conditional) -> <Fetched 304 http://127.0.0.1:8099/page 45 bytes (cache)> attempts=1
404 -> <Fetched 404 http://127.0.0.1:8099/gone 12 bytes> attempts=1 (not retried)
429 then 500 -> <Fetched 0 http://127.0.0.1:8099/page 45 bytes (cache)> attempts=2
after the breaker trips -> <Fetched 0 http://127.0.0.1:8099/page 45 bytes (cache)> attempts=0
slept for : [1.0, 1.0, 1.0, 2.0, 1.0]
log:
('http://127.0.0.1:8099/page', 200, 'ok')
('http://127.0.0.1:8099/page', 304, 'not-modified')
('http://127.0.0.1:8099/gone', 404, 'terminal')
('http://127.0.0.1:8099/page', 429, 'retry in 2.00s')
('http://127.0.0.1:8099/page', 500, 'retry in 0.32s')
('http://127.0.0.1:8099/page', 'gave-up', "<HTTPError 500: 'Internal Server Error'>")
('http://127.0.0.1:8099/page', 'circuit-open', 'circuit open for another 5.0s')Everything above runs on the standard library, deliberately: the virtualenv on this server has no requests, and a scraper with no dependency still runs in three years. Where requests is available the same structure maps onto a Session one-to-one.
When not to do this
If the site has a documented API or a data export, use it. It is faster, stable across redesigns, and it is the answer the owner has already given. A scraper is what you write when that door does not exist.
And if the data is small and you need it once, download it by hand. A scraper is a maintenance commitment: every redesign on the other end is a bug on yours. Ten minutes of clicking beats a week of selector archaeology.
If you do need one that runs unattended, that is automation work we take on, usually alongside the application that consumes the data. Start a project.
We ship and run our own products. We build yours the same way.
Start a project
