Recover Background Jobs After a Restart

From the team behind aidrop.it — one workspace to build, host, and keep changing your code.

Recover background jobs after a restart by recording accepted work outside the worker process, reclaiming interrupted attempts, and making repeated effects safe. Commit the job's obligation with the business record that creates it. Track attempts and completion so a replacement worker can distinguish pending work from work that needs an operator's decision.

In the shop, a deployment replaces the web process after an order commits but before it sends the receipt. Keeping the task in memory leaves the next process without a record that it owes the customer an email. Store that obligation before confirming the order.

Record the order and pending work in one transaction

Insert the order and its pending receipt record in the same database transaction. After the commit, a worker can find the receipt task even if the web process stops. Saving the order and then publishing a job to a separate queue leaves a failure window between the two writes; the application must account for that window.

With a queue, a dispatcher can read the pending record and retry publishing it until the broker accepts it. This is the transactional outbox pattern. A dispatcher may publish twice if it stops before recording acceptance. Carry the same job ID through delivery and design the consumer for that repetition.

For a small workload, a worker can process the database job table without another broker. You still need concurrency control and recovery. Choose a separate queue once the workload justifies operating it.

Give interrupted work a route back to pending

Claim a job with a bounded ownership period, or use a queue library that can recover unacknowledged work after a worker disconnects. Record attempt counts and a deadline for retrying. A process restart must leave another worker a way to reclaim the job; setting a row to running forever creates an obligation that nobody will finish.

For leased jobs, choose an expiry and renewal policy and coordinate claims between workers. A worker can continue after losing its lease, so a lease alone cannot prevent repeated effects. Preserve the job identifier through attempts and check the effect's completion rules.

Retry temporary failures with backoff; cap attempts or elapsed time and retain the final error. Review the oldest pending job's age as well as the count. A running worker can leave one failing receipt behind while processing newer orders.

Account for the effect before acknowledging completion

Acknowledge a job after its business action reaches the outcome you can verify. A worker can send an email and stop before recording completion, so redelivery may send another email. For effects stored in your database, commit the effect and completion together. For an external provider, use its deduplication contract or reconcile uncertain results before repeating them.

A completion flag still leaves a gap between an external call and recording its outcome. Use provider deduplication or document the duplicate risk. The order storage contract separates that pending work from the accepted purchase.

Use the runtime contract to check which storage survives application replacement. Queue persistence preserves data according to its durability settings; your worker code still owns claims, retries, and effects.

A background-job recovery drill

Run this drill with test orders and a test delivery destination. Stop the process at each point, restart it, and inspect the job record and external effect. Record how you resolve an uncertain result. A normal run cannot show whether an interrupted attempt will disappear, become stuck, or repeat its effect.

RECEIPT JOB RECOVERY DRILL
Order ID and stable job ID:
Durable job record or outbox table:
Claim method, expiry, and renewal:
Retry limit, backoff, and failure owner:
External-effect deduplication or reconciliation method:

STOP POINT                      EXPECTED RECOVERY
Before order/job commit         Neither record commits
After commit, before dispatch   Dispatcher finds pending work
After publish, before marking   Repeat delivery keeps the job ID
After claim, before effect      Claim expires or queue reclaims
After effect, before completion Reconcile or deduplicate effect
After retry limit               Keep error; notify named owner

Observed result and remaining duplicate/loss risk:
Oldest pending job age and where the owner checks it:

FAQ

Do background tasks survive a web-process restart? Tasks kept in process memory disappear when you replace that process. A durable job record lets another worker resume work, provided the application also implements claims, retry limits, and a completion policy.

Does a persistent queue guarantee one execution? A worker can repeat a job after losing confirmation of completion. Queue persistence and acknowledgment help recover work, but the application must make the business effect safe to repeat or reconcile the uncertain outcome.

What does aidrop.it provide for background work? aidrop.it offers managed PostgreSQL and a separate persistent Valkey queue within a Project. Your application chooses the job library and implements dispatch, acknowledgment, and recovery. The queue uses append-only persistence with a crash-loss window; it does not promise zero job loss or exactly-once effects.