A customer clicks Cancel. How does that signal reach an ffmpeg process eight function-calls deep?

Our video pipeline does long things. A single job runs face detection at 2 fps over a 10–20 minute video, embeds every detection, clusters them by identity, then renders a focus-cropped output. Two ffmpeg shell-outs and a numpy pass. End-to-end can run several minutes.

When a customer clicks Cancel in the UI, the worker is almost certainly inside subprocess.Popen.wait() for ffmpeg, or inside a numpy loop that doesn't know what HTTP is. The cancel button has to reach across an API boundary, into the worker, down a stack of library calls, and into an OS subprocess — without leaving the job row stuck halfway through a status transition.

The usual answers we rejected:

1. A side-channel via Redis pub/sub. Two sources of truth — the queue says running, Redis says cancelled — and somebody has to reconcile.
2. SIGTERM the whole worker. Every other job in flight on that worker dies with it.
3. Have the handler poll an HTTP endpoint. The worker now depends on the API to know when to stop.

What we landed on is smaller. The jobs table is already the source of truth for job state, so the cancel channel is the jobs table. The API flips status = 'cancelling'. A watcher thread inside the worker process polls the table, finds the cancel, and signals a per-job token. The token has a list of subprocess.Popen handles registered against it; cancelling it SIGTERMs them.

Here's the token, from our worker layer in the deemwar-queue repo:

# src/cancel.py
class CancelToken:
    """Cooperative cancellation signal threaded through one job's execution."""

    def __init__(self):
        self._evt = threading.Event()
        self._lock = threading.Lock()
        self._processes: list[subprocess.Popen] = []

    def is_cancelled(self) -> bool:
        return self._evt.is_set()

    def raise_if_cancelled(self) -> None:
        if self._evt.is_set():
            raise JobCancelledError("job cancelled")

    def register_process(self, p: subprocess.Popen) -> None:
        # Track an ffmpeg/subprocess so cancel() can terminate it.
        ...

    def cancel(self) -> None:
        # Set the signal and terminate every registered subprocess.
        ...

The library threads the token through every long step. The ffmpeg helper calls register_process(proc) before waiting on the Popen, and the per-frame loop calls raise_if_cancelled() between yielded frames. The watcher does one cheap SELECT per tick — WHERE status = 'cancelling' AND job_id = ANY(%s) — against only the jobs currently registered, so polling cost stays constant regardless of total job volume.

Two properties of this design are worth pulling out. The DB as cancel channel adds zero new failure modes — whatever availability story we have for the jobs table is, by construction, the availability story for cancellation. The token is cooperative, not preemptive — the handler picks where it's safe to abort. That sounds weaker than kill -9 but it's stronger: you never leave a half-written manifest, because the only abort points are between safe operations.

The second story lives further down the pipeline. We run Chinese Whispers clustering over face embeddings to group "this is the same person across the video." On busy fixtures — group choreography, school-assembly footage — naive single-pass CW over-merges. Distinct dancers pool into one cluster when their embeddings bridge across simultaneous detections.

The instinct is to tighten the threshold. That over-splits everyone else. The fix is a three-step refinement, and the comment block above the constants is the design doc:

# lib/pipeline.py
# Post-CW refinement: on busy dance/group fixtures CW can pool 5+
# distinct dancers into one cluster when their embeddings bridge.
# (1) Per-timestamp dedup picks the centroid-nearest record when a
# cluster has multiple records at the same sample timestamp; (2) the
# coherence filter drops records whose centre teleports more than
# COHERENCE_MAX_FACE_WIDTHS between consecutive kept records within
# a COHERENCE_MAX_GAP_S window (beyond that gap the subject may have
# walked off and re-entered, so no constraint); (3) a second CW pass
# at a tighter threshold re-clusters each survivor cluster — the
# simultaneous-detection bridges are gone after step (1), so the
# tighter threshold splits the remaining mixed identities without
# the mega-merge that the same threshold would cause on raw input.
COHERENCE_MAX_FACE_WIDTHS = 4.0
COHERENCE_MAX_GAP_S = 1.0
SECOND_PASS_COSINE_THRESHOLD = 0.42

Dedup-per-timestamp removes the simultaneous-detection bridges. The coherence filter drops teleports inside a 1-second window. Then the second CW pass at 0.42 cosine splits the remaining mixed identities. Pipeline reasoning beats parameter tuning — the same 0.42 on raw embeddings would over-split; on records that have already passed the first two filters, it splits exactly the bridges we want.

Both stories collapse to one rule: the system already knows what you need. Don't add a moving part to tell it again. Postgres knows the job is cancelling. The timestamps already encode the temporal-coherence constraint. Lean on what's there.

This is what we're building inside the video product right now. No public sales page yet — engine first, UX next. If long-running, cancellable, identity-aware video work is the shape of your problem, follow our build.

deemwar.com — the company hub.

Questions: io@deemwar.com

— deemwar

References

• Go Authors — Package context (standard library reference) (2024). pkg.go.dev/context
• Sameer Ajmani (The Go Blog) — Go Concurrency Patterns: Context (2014). go.dev/blog/context
• Microsoft Learn — Cancellation in Managed Threads (.NET) (2023). learn.microsoft.com — cancellation-in-managed-threads
• Oracle — Interface Future — cancel(boolean mayInterruptIfRunning) (2014). docs.oracle.com — Future
• PostgreSQL Global Development Group — System Administration Functions: pg_cancel_backend (2024). postgresql.org — functions-admin