A deploy cost us a 58-minute therapy session. Now an interruption costs one stage, not the session.

ClinicLog records a therapy session on the therapist’s phone, turns it into a transcript, and, when the therapist asks, drafts the clinical note from the transcript. A session is an hour of someone’s therapy. It can’t be recorded again, and the therapist has already moved on to the next client. So the one outcome the pipeline can’t afford is losing a session that was recorded well.

On 20 October it lost one.11Commit 7ced7fed7, 2026-10-20: 11 files, +1,567/−283, then five review fixes by the next morning. The loss is recorded in the capture module’s docstring

What a deploy did to a 58-minute session

A routine deploy restarted the background worker in the middle of a pilot session’s transcription. Four reasonable decisions added up to a total loss:

• Our copy of the audio was deleted as soon as it was submitted to the speech recognizer. Less patient audio on our disk is a privacy win.
• The worker deleted the recognizer’s copy on its way out, also for privacy.
• The transcript lived only in the worker’s memory until the final publish.
• The job was allowed one try.

Each of these is defensible alone. Together they meant that the only way back to the session was in the memory of a process we had just restarted.

Three stages, each one written down before the next

The fix splits the job into three stages and saves the point between them before moving on:22workers/session_capture_tasks.py: 2,123 lines; MAX_ATTEMPTS = 5, POLISHED_ATTEMPTS = 3

1. Recognize. Submit the audio and save the recognizer’s job id to the stored result. Only then delete our copy of the audio.
2. Checkpoint. Save the recognizer’s speaker turns before any model touches them. Only then delete the recognizer’s copy.
3. Polish and publish. The readability pass, speaker roles, the split per client, ready.

A run that starts finds whatever the last one left and continues from there. The order is the whole design. Here is the line that matters, with its comment, as it is in the code:

# **Saved before our copy goes**, because from the next line the
# vendor's job is the only copy of the session and this id is how
# a resumed run reaches it. A run cut off between the submit and
# this write resubmits from the audio still on disk, and the job
# it orphaned is the stranded-job sweep's.
result.provider_job_id = job_id
try:
    await _checkpoint(captures, results, result, lease)
except _LeaseLostError:
    # Lost between the check above and the submit. The other run
    # never learns this job id, so nothing would fetch or delete
    # it; it goes now rather than at the stranded sweep.
    await forget_upstream(result)
    raise

# The moment Speechmatics has it, our copy is redundant - on disk
# and in memory both. Deleted here rather than at the end because
# this is the earliest point it can be, and every extra minute is
# a client's therapy hour sitting on a shared volume.
audio_path.unlink(missing_ok=True)

The audio is still deleted at the earliest moment it can be. One privacy rule did change: a run that is cut off no longer deletes the recognizer’s copy, because that copy is what the next run resumes from. A sweep still deletes any job of ours older than six hours. What changed is that something durable exists before each copy goes.

2026-10-20Deploy nowWorker killed

Submit to the recognizerjob id in memory

Delete our audiodeleted

Transcriptworker restarted

Delete the recognizer’s copydeleted on exit

Polish and publishnothing to publish

One try allowed, nothing saved. The session is lost.

A killed worker is the case the queue sees last

A deploy is polite: it sends a stop signal. The worker stops taking new jobs and gives running ones 10 minutes to finish, and the deploy restarts the two workers one at a time.33Workers drain for 600 s inside an 11-minute stop grace period; the deploy restarts the two workers one at a time A job cut off past that is cancelled, saves its checkpoint, and is run again.

A worker killed outright (out of memory, a host that crashes) gets no signal. Our job queue keeps treating its job as in progress until a marker expires, and that marker lives as long as the longest job timeout on the worker, not this job’s: a whole morning’s notes may run 2.4 hours, measured. For that long the therapist would see the session as processing, with nobody working on it.

So each run now holds a lease: a short-lived key with a 60-second lifetime, renewed every 15 seconds. A killed worker stops renewing it. A sweep runs every minute and queues again any session that still has work outstanding, no lease, and no write for 3 minutes.44verbatim/capturelease.py: 60 s lease, 15 s heartbeat; sweep every minute, re-queues after 180 s with no lease and no write Queuing twice is safe because every run takes the lease first and starts from the checkpoint. A duplicate that arrives behind a live run does nothing.

In the end-to-end run I did before shipping the fix, on isolated workers with synthetic audio, a worker killed with SIGKILL had its session back on the queue in 3.5 minutes. The queue’s own marker still had 2.4 hours left.

a killed run back on the queue, in the end-to-end run before the fix shipped

3.5 min

what the queue alone would have waited

2.4 h

Retrying forever is its own failure

A run that keeps dying can’t be retried forever. A session gets 5 runs, then the therapist is told it was interrupted and the saved transcript is deleted. The model passes are the part that runs for minutes, so they are the likeliest part to keep getting cut off. Past the third run they are skipped and the transcript is published as the recognizer heard it. Three, because one deploy can cancel the same job twice: each worker drains in turn, and the second can pick up what the first let go. A transcript without polish is worth more than no transcript. The first version of the fix let a saved transcript lift the 5-run limit; review caught it within the hour.

The pipeline’s two test files run 100 tests, and the resume, lease and sweep cases are among them.55100 tests in test_session_capture_tasks.py and test_capturelease.py, passing on main (2026-11-01)

What I’d keep for any client

• Write the state down before you delete the copy. Every delete in a pipeline should come after something durable that replaces it.
• Privacy and durability are separate decisions. Deleting early was right. Deleting before anything was saved was the bug.
• Your queue sees a dead worker only when its longest timeout runs out. A lease with a heartbeat sees it in a minute. Make re-queuing safe and let a sweep do it.
• Bound the retries, then degrade. After a few tries, ship the plain result instead of nothing.