Build Unbreakable Workflows! (feat. Temporal)

It's Thursday afternoon. Friday's a holiday and the weekend already feels like it's started.

There's a movie with friends tonight. Tomorrow morning, you’re all hitting the road. Then there’s poker, trekking, rafting and somewhere in between all of it, a quiet weekend at a resort tucked deep inside a wildlife sanctuary.

For once, everything is lined up perfectly. No deadlines. No late-night incidents. Nothing urgent waiting for you. Just a long weekend waiting for you.

You can almost feel yourself switching off.

Then a fellow engineer walks over.

"Hey, our bread-and-butter workflow is failing. Customers are reporting it, Support is overwhelmed and it's been escalated to us."

Fine. Probably a minor bug.

You start digging through incomplete records.

Boom. The server crashes.

Now you have three problems: the workflow is broken, the server is crashing and you have incomplete workflows that need reconciliation based on the stage it is stuck in.

Your movie is gone. The road trip hangs by a thread. Nobody knows the actual root cause. At this point, you start praying to God.

This article is about building systems resilient enough that your server can have a bad day without ruining your weekend.

The least your critical workflows deserve are observability, resiliency and durability.

One tool I've been using for a while that brings all three together is Temporal.

Temporal is a durable workflow orchestration platform. Think of it as a system that lets you run long-running, reliable business processes while handling many of the resiliency patterns you'd otherwise have to build yourself.

At a high level, a Temporal Workflow defines the business process, while Activities perform the actual work.

A Workflow is the durable orchestration logic. It describes what needs to happen, in what order and under what conditions. Temporal persists its execution history, so if a Worker crashes, the Workflow can be reconstructed and continue from where it left off.

An Activity is where the actual side effects and external interactions happen - calling an API, writing to a database, sending an email, charging a customer, generating a report, etc. Activities are designed to be retried when they fail.

These are the benefits I've personally seen from using Temporal. There's a lot more it can do, but these are the features that really hit the spot for me.

1. At-least once Activity execution
   1. Activities can execute more than once when failures occur.
   2. That means your Activity code should be designed to be idempotent - running it twice should not produce an unintended second side effect.
2. Workflow-level deduplication
   1. You can use a stable Workflow ID to prevent duplicate workflows from being started for the same business operation.
   2. If the same request arrives twice, Temporal can ensure you don't accidentally run two copies of the workflow.
3. Timeout
   1. Workflows and Activities can have time limits.
   2. If something gets stuck waiting on a downstream service, you don't end up with an indefinitely hanging operation.
   3. You can also set timeout for an external signal, ensuring the Workflow eventually times out if the signal never arrives.
4. Automatic retries with backoff and jitter
   1. If an Activity fails because of a transient problem, Temporal can automatically retry it.
   2. You can configure increasing delays between attempts, while jitter adds randomness to those delays.
   3. This prevents large numbers of failed Activities from retrying at exactly the same time, helping avoid the thundering herd problem where a recovering downstream service is suddenly overwhelmed by a flood of retries.
5. Crash recovery
   1. The Worker can crash, the machine can restart or a deployment can happen.
   2. The Workflow itself doesn't disappear with the process that was executing it. A new Worker can pick it back up.
6. Durable state and checkpoints
   1. Temporal persists Workflow execution history independently of the Worker executing it.
   2. If a Worker crashes or restarts halfway through a Workflow, another Worker, even on a completely different server, can replay the persisted history, reconstruct the Workflow state and continue execution from where it left off.
   3. You can also control which state should be persisted as part of the Workflow versus what can be recomputed or fetched again when needed.
7. Compensation
   1. Not every failed operation can simply be retried forward.
   2. Sometimes you need to undo a previous successful operation.
   3. If an activity fails, you can add the logic to compensate it. This is not a straight forward configuration, but can be implemented quiet easily.
8. Child workflows
   1. A Workflow can start another Workflow.
   2. This lets you break large business processes into smaller, independently managed processes without putting everything into one giant Workflow.
9. Workload isolation and load distribution
   1. Temporal distributes Workflow and Activity execution across available Workers through Task Queues, while allowing you to isolate different workloads across separate Worker capacity.
   2. This prevents a problematic workload or traffic spike from consuming all your resources and affecting critical Workflows.
   3. As demand increases, you can add more Workers to scale horizontally without changing your business logic.
10. Run concurrent activities
    1. Temporal lets you start multiple Activities concurrently and wait for all of their results before continuing the Workflow.
    2. You can choose which Activities should execute sequentially and which independent Activities can be executed concurrently, allowing you to reduce overall Workflow execution time.
    3. Temporal also lets you control the maximum number of Activities executing at the same time. For example, if you have 10,000 Activities to process but only want 20 running concurrently. As an Activity completes, the next one can start.
11. Workflow coordination
    1. Workflows can coordinate with other Workflows using Temporal's built-in communication mechanisms, such as Signals, Queries and Updates.
    2. This allows Workflows to exchange information, respond to events and coordinate state without having to build custom coordination logic around databases, queues or polling.
12. Downstream protection
    1. Temporal can help control how quickly work is executed through Task Queue and Worker-level concurrency/rate controls.
    2. So if your downstream API says "maximum 100 requests/minute", you can design your Workers and Task Queues so that you don't blindly hammer it during a retry storm.
13. Human in the loop
    1. A Workflow doesn't have to finish immediately.
    2. It can remain durably paused while waiting for a human decision or approval, without requiring you to maintain separate state in a database.
    3. Once the decision is made, the human can signal the Workflow, allowing it to resume and continue execution from where it was waiting.
14. Observability and operational recovery
    1. Temporal provides visibility into Workflow and Activity execution, including what ran, what succeeded, what failed, what was retried and where a Workflow is currently stuck.
    2. When work repeatedly fails and shouldn't be retried forever, you can inspect the failed execution and intervene manually rather than building a separate dead-letter queue and reconciliation system.
    3. Temporal's UI also provides operational controls to retry, cancel, terminate or reset Workflows when needed.
15. Scheduled execution
    1. Workflows can be scheduled to run at a specific time or on a recurring schedule.
    2. This is useful for periodic business processes such as daily reports, billing, data processing, reminders or cleanup jobs.
    3. For example, if a notification needs to be sent a few minutes before a meeting starts, you can schedule the Workflow to execute at the required time.
    4. Temporal manages the schedule and Workflow execution for you, so you don't need to build your own scheduler or rely on a server's cron job.

NOTE: Please don't route all your transactions through Temporal. Temporal adds infrastructure, operational overhead and application boilerplate, so it isn't worth introducing for every operation. Use it when a business process genuinely benefits from durable execution, retries, timeouts and orchestration. For simple, short-lived operations, basic asynchronous processing or high-volume stateless workloads, a database, queue or streaming platform may be a better fit.

Hope you get to enjoy all those weekend plans! :)

If you've built reliable workflows in production, with or without Temporal, I'd love to hear what has worked for you. What resiliency feature, pattern or practice has saved you from a production headache?

And if you use Temporal, let me know if I missed any feature or pattern that deserves to be on this list. There’s a lot more to Temporal than what I’ve covered here and I’d love to hear what has made the biggest difference for you.

May your Workflows stay healthy, your Workers stay up and your weekends stay uninterrupted.