Most Node.js tutorials treat error handling as an afterthought. You wrap a route handler in a try/catch block, log the stack trace, and return a 500. That mindset survives because a real person is waiting on the other end of an HTTP request. Background jobs are different. In a queue system, there is no impatient client to reply to, no automatic browser refresh. There is only a worker, a payload, and a retry counter ticking upward in silence. When things go wrong, they go wrong slowly, then all at once. One misclassified error can stall an entire pipeline or page an engineer at three in the morning.
The disconnect is simple. Request-response cycles fail fast and fail loud. A queue failure is quiet. A worker might churn through hundreds of jobs before a database connection drops. Without clear handling rules, the worker retries instantly, hammers the already struggling database, and crashes. Because no one is watching the worker directly, the first sign of trouble is often a cascading backup or a disk full of log files. You need more than catch blocks. You need a strategy that treats different failures differently and protects the rest of the system from a single bad job.
Two Kinds of Failure
Start by splitting every error into one of two buckets.
Retryable errors are transient. A network timeout to a third-party API, a 429 rate-limit response, or a temporary database replica lagging behind its primary. These are symptoms of pressure, not bugs. The system might heal itself in thirty seconds. Retryable jobs deserve another shot, but only under controlled conditions.
Permanent errors are mistakes. Invalid JSON in the payload, a missing user ID, a required file that does not exist in storage. These will fail on the hundredth attempt exactly as they failed on the first. Retrying them burns CPU cycles, wastes queue slots, and creates toxic back-pressure that delays healthy jobs. The only useful place for a permanent failure is a log, an alert, or a dead-letter queue. It does not belong in the retry loop.
Build a Decision Engine
Classify immediately. Do not leave this decision to the queue framework. The moment you catch an error, decide its fate.
In practice, this means creating custom error classes or wrapper functions that inspect the failure before bubbling it up. If a database driver throws a connection reset, your handler should tag that as retryable. If a payload validator throws a schema mismatch, tag it as permanent. Many job processors default to retrying everything, which is the most expensive choice you can make. Reject permanent jobs instantly. Either drop them or reroute them to a dead-letter queue where they cannot poison the main pipeline. This single habit prevents snowball effects more reliably than any infrastructure change.
Back Off, But Smarter
When you do retry, never do it immediately. If a database is down, a burst of workers hammering it every second looks like a denial-of-service attack from the inside. Use exponential backoff. Wait one minute, then five, then fifteen. Give the upstream system room to recover.
But exponential backoff alone is not enough. If a thousand jobs fail at the same time because a service restarted, their retry schedules will align. They will hit that service simultaneously when it comes back online, potentially knocking it down again. Add jitter: a small random offset to each delay. Scattering retries across a few seconds prevents synchronized stampedes. The math is simple, but the stability it buys is enormous.
Preserve the Evidence
A dead-letter queue is your audit trail, not a trash bin. When a job exhausts its final retry, do not simply delete it. Move the entire payload, along with the error context and the retry history, to a DLQ.
This preserves evidence. A human can inspect the job, patch the bug, and replay it manually if necessary. More importantly, monitor the depth of your DLQ. A sudden spike in dead-lettered jobs is often the earliest warning of a bad deploy, a schema change gone wrong, or an external vendor breaking their contract. Treat DLQ growth as a leading indicator, not a trailing one. If your DLQ is filling up, something upstream has changed and your team needs to know before the backlog spreads.
Design for the Retry
Design every job as if it will run twice, because it might. A worker can fail halfway through processing, get rescheduled, and execute again. If your job charges a customer, sends an email, or increments an inventory count, a naive retry creates duplicates.
The fix is idempotency. Before you perform a side effect, check whether it already happened. Use a unique identifier from the job payload as an idempotency key. Store that key in a short-lived cache or a database table with a uniqueness constraint. If the key exists, skip the work and return success. This turns retries from a risk into a harmless no-op. It takes a few extra lines of code, but it saves you from explaining to finance why revenue doubled overnight.
Protect the Process
Unbounded promise rejections and stray exceptions can kill a Node.js process without warning. In a worker, that means dropped jobs and an orchestrator scrambling to restart the container.
Register global handlers for unhandledRejection and uncaughtException. Their job is not to rescue the application. It is to perform the minimum cleanup necessary, then exit. Let Docker, Kubernetes, or systemd restart the worker with a clean memory state. Limping along after a global handler fires invites memory leaks and corrupted state. A fast, clean death is safer than a slow zombie process that processes jobs incorrectly. Trust your orchestrator to bring you back; do not try to outsmart a corrupted runtime.
Respect the Signal
Workers get shut down during deployments, scaling events, and node rotations. If your process dies the instant it receives SIGTERM, you abort whatever job is currently in flight. That job may never finish, and its retry counter might not even have incremented yet.
Listen for SIGTERM and SIGINT. When a signal arrives, stop pulling new jobs from the queue. Finish the current job if you can. Set a hard timeout, perhaps thirty seconds, after which you exit regardless. This graceful shutdown respects the queue and avoids false failures. Your deployment pipeline should treat a worker that exits cleanly as healthy, while a crashed worker should trigger an alert.
The Real Takeaway
Reliable queue handling is not about catching every error. It is about making deliberate decisions for each failure mode. Retry the transient ones with patience. Bury the permanent ones quickly. Shield your workers from stampedes, protect your data with idempotency keys, and let dying processes exit cleanly. When every failure has a defined path, three in the morning becomes just another hour. Your pipeline keeps moving, and your team keeps sleeping.