We deleted the worker-watchdog container. Postgres does that job now.

In reqsume we run long-running jobs through a per-action queue — resume rewrites, PDF builds, AI chat iterations. Each job claims a row in a jobs table, sets status = 'running', does the work, then sets status = 'done' or 'failed'. Standard stuff.

The problem we kept hitting: workers sometimes die mid-job. Process gets OOM-killed, container restarts, network blip during a long ffmpeg shell-out. The row stays in 'running' forever. Customers stare at a spinner. We get a support ticket two hours later.

The naive read is "this is what message queues are for — the visibility timeout will re-deliver the message and another worker will pick it up." We were already on pgmq, which has exactly that mechanism. It didn't help. Visibility-timeout redelivery only works if the worker never acknowledged the message — but our workers ack the queue first, then write the running row. By the time the worker dies mid-job, pgmq already considers the message handled. The job row is the source of truth, and the job row is now an orphan.

So we needed a janitor: something that scans the jobs table every few minutes and fails rows that have been stuck in 'running' for too long.

We considered three obvious answers:

1. Add a sidecar container that runs a watchdog loop. That's one more thing to deploy, one more thing to restart, one more thing that can be the silent point of failure when the watchdog itself dies.
2. A Kubernetes CronJob. Except reqsume doesn't run on Kubernetes — we run on a shared box with systemd and a couple of Go binaries. Pulling in k8s for one cron tick would be absurd.
3. A goroutine inside the API itself. Tempting, but then the watchdog dies with the API, and if the API is the thing that's wedged, the watchdog can't fire.

The answer we landed on: pg_cron. Postgres extension, ships with most managed Postgres providers (RDS, Supabase, plain Hetzner Postgres), turns the database itself into the scheduler. The watchdog isn't a process — it's a SQL statement registered with the DB at boot. If the API crashes, the watchdog keeps firing. If a worker crashes, the watchdog cleans up after it. One container fewer to run.

Here's the registration code from apps/api/src/appinitializer/jobs_cron.go:

const (
    jobsStuckWatchdogSchedule = "*/5 * * * *"
    jobsStuckWatchdogCommand  = `
        UPDATE jobs
        SET status = 'failed',
            error  = jsonb_build_object(
                       'code','watchdog_timeout',
                       'message','worker timed out (running >10 min)'),
            completed_at = NOW()
        WHERE status = 'running'
          AND started_at < NOW() - INTERVAL '10 minutes'`

    jobsDailyCleanupSchedule = "0 2 * * *"
    jobsDailyCleanupCommand  = `
        DELETE FROM jobs
        WHERE status IN ('done','failed')
          AND completed_at IS NOT NULL
          AND completed_at < NOW() - INTERVAL '24 hours'`
)

BootJobsCronSchedules walks those tuples at startup and calls cron.schedule(name, schedule, command). Re-registering the same name updates the schedule in place — so the same boot path works for first install, redeploy, and schedule changes.

Two small details we'd flag for anyone copying this pattern:

The error is jsonb, not text. We write jsonb_build_object('code','watchdog_timeout', ...) instead of a string like "worker timed out after 10m". That single change is what lets our alerting code stay simple: it reads error->>'code' and routes — watchdog_timeout pages an engineer, validation_failed doesn't. Free-text error messages couldn't be classified without regex matching that breaks the moment someone reword the message.

The boot path tolerates failure. If pg_cron isn't installed or the registration call fails, we log.Printf and continue. The API still serves traffic; the janitor just doesn't run. An operator can rerun boot to retry. The alternative — crashing the API because we couldn't register a 5-minute cron — would have been worse than the problem we were solving.

After the watchdog runs, the row looks like this:

id     | 8e3f2c1a-...
status | failed
error  | {"code":"watchdog_timeout","message":"worker timed out (running >10 min)"}
completed_at | 2026-05-21 09:42:00+00

Downstream code that polls GET /api/jobs/:id sees status: failed, stops polling (no Retry-After on terminal states), and surfaces the error to the user.

A portability note: pg_cron isn't installed on every Postgres. RDS requires you to add it as a parameter-group extension. Plain self-hosted Postgres needs CREATE EXTENSION pg_cron and a shared_preload_libraries line in postgresql.conf. If you can't get the extension, the same pattern works with a tiny shell loop and psql — you just give up the "DB-as-scheduler" property and you're back to a process to babysit.

If you're already paying for a managed Postgres, you might already have a scheduler you're not using.

This pattern lives in reqsume's job pipeline today — the worker layer behind every resume-rewrite, chat-iterate, and PDF-download action. We're building reqsume to make AI-driven resume editing feel like a desktop app even when half the work is happening on a server. Less moving parts in the worker layer means fewer 3 AM pages.

reqsume.com — see it in action.

Questions: io@deemwar.com

— deemwar

References

• Citus Data — pg_cron: run periodic jobs in PostgreSQL (2016–present). citusdata/pg_cron on GitHub
• Tembo — pgmq: a lightweight message queue built on Postgres (2023–present). tembo-io/pgmq on GitHub
• Tembo — Introducing PGMQ: Simple Message Queues built on Postgres (2023). tembo.io/blog/introducing-pgmq
• Supabase — pg_cron extension docs (scheduling jobs inside Postgres). supabase.com/docs/guides/database/extensions/pg_cron
• Supabase — Queues: durable message queues with guaranteed delivery, built on Postgres. supabase.com/docs/guides/queues