All articles

Laravel 9 min read AgiCAD studio

Sixteen Queue Workers, One Job: When retry_after Is Shorter Than the Job

A scraping queue that would not drain with sixteen workers, because retry_after was 90 seconds and the jobs ran for two hours. Every idle worker re-reserved the same job.

A crowd of worker robots all grabbing the same glowing parcel while an hourglass beside them has run out

On CijeneHR, one of our products and a price-comparison site that scrapes retailer catalogues, the scraping queue stopped draining. Sixteen workers were running. The backlog grew for weeks. Every instinct said "the scraper is slow, add capacity", and adding capacity was exactly the wrong move: it made the problem worse in a way that is invisible until you look at the right table.

The symptom

The queue depth went up and to the right. Workers were busy - ps showed sixteen live queue:work processes, all consuming CPU. The log showed scrapes starting, and also MaxAttemptsExceeded exceptions we had filed away as "some retailer is flaky", because the failures were sporadic and the job in question always seemed to have finished successfully somewhere.

That last detail was the clue we ignored for too long: the job completed, and then failed.

The wrong hypothesis

Our first two theories were the obvious ones, and both were wrong in the same way: they assumed throughput was a capacity problem.

First, not enough workers. We had already gone from four to sixteen and the backlog grew faster afterwards - which should have stopped us, because nothing about "not enough workers" predicts that quadrupling them makes it worse. Second, the scraper is slow. It was slow, around two hours per retailer catalogue, but slow is not stuck: two-hour jobs with sixteen workers should still clear sixteen catalogues every two hours. It was clearing roughly one.

That ratio is the tell. Throughput of one job per job-duration regardless of worker count does not describe a slow system. It describes sixteen workers doing the same piece of work.

The query that proved it

The database driver keeps everything in a jobs table, so the evidence is right there in SQL. Reserved jobs carry a reserved_at timestamp and an attempts counter.

SELECT reserved_at, COUNT(*) AS rows_reserved, MAX(attempts) AS max_attempts
  FROM jobs
 WHERE reserved_at IS NOT NULL
 GROUP BY reserved_at
 ORDER BY rows_reserved DESC;

A healthy queue gives you a scatter of distinct reserved_at values with attempts = 1. What we got was clusters - a pile of rows sharing one reserved_at second, with attempts climbing well past one:

(1761500000, 16, 16)
(1761500090, 1, 1)

Sixteen rows re-reserved in the same second, an attempt counter at sixteen. That is not a slow scraper. That is the queue handing the same work out over and over.

The cause

The database driver does not have a per-message visibility timeout negotiated with the worker. It has a single connection-level setting, retry_after, which says: if a job has been reserved for longer than this many seconds, assume the worker died and let someone else take it. The default is 90 seconds, because the default is tuned for jobs that send an email.

Our jobs declared public $timeout = 3600; and ran for about two hours. retry_after in config/queue.php was still the shipped default. So every 90 seconds, an idle worker looked at the table, saw a job reserved more than 90 seconds ago, concluded the worker holding it must be dead, and took it. Ninety seconds later, so did the next one. Within about twenty minutes all sixteen workers were scraping the same retailer, and when the original worker finally finished, the other fifteen came back to a job whose attempt counter had run past $tries and threw MaxAttemptsExceeded.

Laravel's documentation states the rule plainly: retry_after must be greater than the longest time any job takes. It is easy to miss on the database driver specifically, because the value lives in the connection config rather than next to the job, and because the default is silently fine right up until you write your first long job.

Proving it without waiting two hours

We do not like fixing a config value on a theory, so we rebuilt the mechanism against sqlite: a jobs-shaped table, a virtual clock, sixteen workers, one job taking 3,000 seconds.

<?php
declare(strict_types=1);

$db = new PDO('sqlite::memory:');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$db->exec('CREATE TABLE jobs (
    id INTEGER PRIMARY KEY,
    queue TEXT NOT NULL,
    attempts INTEGER NOT NULL DEFAULT 0,
    reserved_at INTEGER NULL,
    available_at INTEGER NOT NULL
)');

function simulate(PDO $db, int $retryAfter, int $duration, int $workers, int $horizon): array
{
    $db->exec('DELETE FROM jobs');
    $db->exec("INSERT INTO jobs (id, queue, attempts, reserved_at, available_at)
               VALUES (1, 'scraping', 0, NULL, 0)");

    $busyUntil = array_fill(0, $workers, 0);   // when each worker is free again
    $log = [];

    for ($now = 0; $now <= $horizon; $now++) {
        for ($w = 0; $w < $workers; $w++) {
            if ($busyUntil[$w] > $now) {
                continue;                       // still running something
            }
            // This is what the database driver does: take a job that is either
            // unreserved, or reserved longer than retry_after ago.
            $sel = $db->prepare(
                'SELECT id, attempts FROM jobs
                  WHERE queue = ?
                    AND available_at <= ?
                    AND (reserved_at IS NULL OR reserved_at <= ?)
                  LIMIT 1'
            );
            $sel->execute(['scraping', $now, $now - $retryAfter]);
            $job = $sel->fetch(PDO::FETCH_ASSOC);
            if (!$job) {
                continue;
            }
            $db->prepare('UPDATE jobs SET reserved_at = ?, attempts = attempts + 1 WHERE id = ?')
               ->execute([$now, $job['id']]);
            $busyUntil[$w] = $now + $duration;
            $log[] = sprintf('t=%-5d worker %-2d reserved job %d (attempt %d)',
                $now, $w + 1, $job['id'], $job['attempts'] + 1);
        }
    }

    return [$db->query('SELECT attempts FROM jobs WHERE id = 1')->fetchColumn(), $log];
}

foreach ([90 => 'default retry_after', 3900 => 'retry_after > timeout'] as $retryAfter => $label) {
    [$attempts, $log] = simulate($db, $retryAfter, 3000, 16, 2000);
    printf("--- retry_after=%d (%s) ---\n", $retryAfter, $label);
    foreach (array_slice($log, 0, 4) as $line) {
        echo $line, "\n";
    }
    if (count($log) > 4) {
        echo '    ... ', count($log) - 4, " more reservations\n";
    }
    printf("total reservations of job 1 in 2000s: %d\n\n", $attempts);
}
--- retry_after=90 (default retry_after) ---
t=0     worker 1  reserved job 1 (attempt 1)
t=90    worker 2  reserved job 1 (attempt 2)
t=180   worker 3  reserved job 1 (attempt 3)
t=270   worker 4  reserved job 1 (attempt 4)
    ... 12 more reservations
total reservations of job 1 in 2000s: 16

--- retry_after=3900 (retry_after > timeout) ---
t=0     worker 1  reserved job 1 (attempt 1)
total reservations of job 1 in 2000s: 1

Sixteen reservations against one - a simulation we trust more than any amount of reasoning about what the driver "probably" does.

The fix

One number, in the connection config rather than in the job:

<?php

return [
    'default' => env('QUEUE_CONNECTION', 'database'),

    'connections' => [
        'database' => [
            'driver' => 'database',
            'connection' => env('DB_QUEUE_CONNECTION'),
            'table' => env('DB_QUEUE_TABLE', 'jobs'),
            'queue' => env('DB_QUEUE', 'default'),
            // MUST be greater than the longest job's $timeout.
            'retry_after' => (int) env('DB_QUEUE_RETRY_AFTER', 90),
            'after_commit' => false,
        ],
    ],
];
DB_QUEUE_RETRY_AFTER=3900

Three thousand nine hundred against a $timeout of 3,600 - above it, with headroom for the worker's shutdown handling. The job states what it expects, so the next person sees both halves of the contract:

<?php

namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\Middleware\WithoutOverlapping;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;
use Throwable;

class ScrapeRetailer implements ShouldQueue
{
    use Dispatchable;
    use InteractsWithQueue;
    use Queueable;
    use SerializesModels;

    public int $timeout = 3600;
    public int $tries = 3;
    public bool $failOnTimeout = true;

    public function __construct(public readonly int $retailerId) {}

    public function backoff(): array
    {
        return [60, 300, 900];
    }

    public function retryUntil(): \DateTimeInterface
    {
        return now()->addHours(12);
    }

    public function middleware(): array
    {
        return [(new WithoutOverlapping((string) $this->retailerId))->expireAfter(3700)];
    }

    public function handle(): void
    {
        // ... the long-running scrape
    }

    public function failed(?Throwable $e): void
    {
        Log::error('scrape failed permanently', [
            'retailer_id' => $this->retailerId,
            'reason' => $e?->getMessage(),
        ]);
    }
}

backoff() returning an array gives a different delay per attempt, and retryUntil() overrides $tries entirely - with both set, the deadline wins. For a scrape whose value decays, a wall-clock deadline is more honest than a count.

What the locking helpers do and do not protect. ShouldBeUnique stops a second job with the same key being dispatched while one is pending or running. WithoutOverlapping is middleware: the job is dispatched normally, but a second one that acquires no lock is released back to the queue instead of running concurrently. Neither helps with the retry_after bug, because there the queue is re-reserving the same row - from the lock's point of view it is the same job, and in some configurations the same lock owner. And both are built on the cache lock, so their guarantees are only as strong as the cache store you point them at: on the array or file driver across multiple servers, they guarantee nothing at all. Point them at Redis or a database lock store, and set expireAfter above your longest job or a crashed worker will hold the lock forever.

The wider lesson: make the job idempotent

The config fix stops this particular duplication. It does not make duplication impossible, and treating it as if it did is the real mistake. Every queue worth using delivers at least once, not exactly once: a worker can be OOM-killed after calling a third-party API and before marking the job complete, a deploy can restart workers mid-job, the network can eat the acknowledgement. Any job that calls an external API will run twice eventually, and if that call charges a card or posts a price update, running twice is a bug with a cost.

The pattern that actually makes retries safe is an idempotency key: derive a stable key from the payload, record it under a unique constraint before the call, and treat the duplicate-key violation as success.

<?php

namespace App\Services;

use Illuminate\Database\QueryException;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Http;

class PriceFeedClient
{
    public function publish(array $payload): array
    {
        $key = hash('sha256', json_encode($payload, JSON_THROW_ON_ERROR));

        try {
            DB::table('idempotency_keys')->insert([
                'idempotency_key' => $key,
                'created_at' => now(),
            ]);
        } catch (QueryException $e) {
            if ($e->getCode() === '23000') {
                // Someone already did this. That is success, not an error.
                $row = DB::table('idempotency_keys')->where('idempotency_key', $key)->first();

                return ['replayed' => true, 'response' => json_decode($row->response ?? 'null', true)];
            }
            throw $e;
        }

        $response = Http::withHeaders(['Idempotency-Key' => $key])
            ->post('https://api.example.com/v1/prices', $payload)
            ->throw()
            ->json();

        DB::table('idempotency_keys')
            ->where('idempotency_key', $key)
            ->update(['response' => json_encode($response)]);

        return ['replayed' => false, 'response' => $response];
    }
}

The unique constraint is the whole mechanism, so it has to exist in the schema, not in your head:

Schema::create('idempotency_keys', function (Blueprint $table) {
    $table->id();
    $table->string('idempotency_key', 64)->unique();
    $table->json('response')->nullable();
    $table->timestamp('created_at')->nullable();
});

We ran the same logic in the sqlite harness, calling it twice with an identical payload:

attempt 1: called the API (9d7aefcd166755ec2e4aa5374023ebc9d0596df7e7f88e606c5f6ba49bb4889a)
attempt 2: already done (9d7aefcd166755ec2e4aa5374023ebc9d0596df7e7f88e606c5f6ba49bb4889a)
rows in idempotency_keys: 1

The second attempt is absorbed by the database rather than an application-level check, which matters: a SELECT-then-INSERT has a race window and a unique constraint does not. Note what the key derives from - the payload, not a UUID generated at dispatch, because a retried job must produce the same key. Send it to the vendor as an Idempotency-Key header too where supported. This kind of durability work is part of our backend application projects and services, and it is cheaper to build in than to retrofit after a duplicate-charge incident.

When not to do this

If your jobs are short - a mail send, a thumbnail, a webhook relay - leave retry_after at 90. The default is correct for the workload it was chosen for, and raising it has a real cost: it is also how long the queue waits before recovering a job from a genuinely dead worker. Set it to an hour on a mail queue and a crashed worker strands those messages for an hour.

And if you have long jobs and short jobs, the answer is not a compromise number. Give them separate connections with separate retry_after values and separate workers. If you find yourself splitting the difference, that is the config telling you the queues should have been split.

If a queue in your application is not draining and the worker count has stopped helping, the jobs table usually knows why. Start a project with the output of that query and we will tell you what it is looking at.

We ship and run our own products. We build yours the same way.

Start a project

Join the article list

We are collecting addresses now and will email new articles once the newsletter starts. No spam, unsubscribe any time.

Hour Blocks from $180reply within two business days Start a project