All articles
About 16 min read

Building durable agents with Temporal

tl;dr

Building reliable AI agents means handling failures, running tasks in parallel, waiting for human approval, and keeping long-running work alive. This article shares what I’ve learned about using Temporal for these tasks, with practical Go examples. It also covers worker discovery with Kyverno and scaling workers with KEDA to reduce idle resource use, along with the trade-offs to consider.

Let's start from the beginning

Distributed systems are not new. Early discussions go back to the mid-1960s. For example, Paul Baran’s paper On Distributed Communications explores distributed network communication, redundancy, and failover. Dijkstra's work on mutual exclusion also helped shape how we think about concurrency and coordinating processes.

Over time, these ideas developed into theory and then into practical systems. One example is the Network File System, released by Sun Microsystems in 1984. Systems like NFS brought challenges such as consistency, write durability, performance, and failure recovery into everyday use.

The need to solve these problems only grew as cloud services became the standard way to run modern systems. Today, it is relatively easy to set up resources across regions, but distributed transactions, locks, process synchronisation, and many other challenges remain.

Agentic era

For better or worse, we are entering a new chapter where personalised AI assistants, agentic flows, and automations play a growing part in our daily lives, work, and entertainment. I look at these changes mainly from an engineering perspective. The pace is extreme. We have quickly moved from “Ask on Stack Overflow” to “Ask ChatGPT” to “Run agentic loops”. Now we are building personalised AI assistants, harnesses, and remote coding agent platforms.

Building these systems well is not easy. Some agents run for hours, some actions need manual approval, and agent failures or network issues need to be handled gracefully. Preserving history and context also matters for reliability, token usage, and cost. Familiar distributed systems patterns, such as child workflows, parallel execution, and long-running activities, fit naturally into agentic systems. They are building blocks for durable workflows and what are often called durable agents.

This is how Restate, a durable runtime for agents, describes a durable workflow:

A durable workflow is a programming paradigm and architecture that lets long-running code execute reliably across crashes, server restarts, network failures, or human-in-the-loop pauses without losing its state or starting over from scratch

Durable workflows

There are many tools for building durable workflows, including workflows for agentic systems. The list keeps growing, and their capabilities keep changing. A few I have looked into are Apache Airflow, Temporal, Vercel Workflows, and Cloudflare Workflows. Each has its own features and trade-offs. Some work best within a specific ecosystem, such as Vercel or Cloudflare, while others offer self-hosting options and fit into existing infrastructure.

Most of my experience with durable workflows comes from using Temporal at Miro for a range of use cases. Along the way, I have learned a lot about its patterns and possible optimisations. Temporal is also used by other companies building agentic systems. For example, Lee Robinson highlights its use for Grok Bot, and Temporal offers an integration with OpenAI’s Agents SDK.

The Temporal documentation covers a wide range of patterns. I’ll focus on those commonly used in agentic systems and those I’ve worked with firsthand. For each pattern, I’ll link to a real implementation and explain its use case.


Child Workflows

Child Workflows split complex business logic into smaller, reusable units. Each child runs as an independent Workflow with its own ID, event history, and lifecycle. This gives you isolation, independent tracking, queries, timeouts, retries, and the option for a child to outlive its parent. Child Workflows can run synchronously, asynchronously, or only when certain conditions are met. This makes them a useful pattern for a parent agent working with sub-agents.

Agentic platforms use child workflows for several tasks. A common example is a parent agent coordinating sub-agents, each with its own context window. A session workflow can also start a SandboxLifecycle child workflow to provision a VM, keep it running, and tear it down. A PullRequest child workflow can wait for CI results and review comments even after the chat session ends.

Practical example: A user uploads a PDF and asks, “What are the main points in this document?” The conversation workflow handles the request and starts a DocumentAnalysisWorkflow child workflow. The child prepares and analyzes the document, then returns a result the main agent can use in its reply.

The child has its own workflow history, task queue, timeout, and processing budget, so progress and failures can be tracked separately from the conversation workflow.

One task, its own workflowStatic view
The conversation starts a document-analysis child with its own history, task queue, and execution limit. The child prepares and analyzes the document, then returns a result so the conversation can reply.
for _, a := range s.Batch {
	if a.Outcome == "authorized" {
		if a.Scope.ToolName == "file_upload" {
			// Pass the approved scope and processing budget to the child.
			in := chat.DocumentAnalysisInput{
				Scope:  a.Scope,
				Model:  model,
				Budget: chat.DefaultDocumentBudget(),
			}
 
			// Give the child its own ID, task queue, and execution limit.
			child := workflow.WithChildOptions(
				turnCtx,
				workflow.ChildWorkflowOptions{
					WorkflowID:               in.ChildID(),
					TaskQueue:                WebDocumentsQueue,
					WorkflowExecutionTimeout: 15 * time.Minute,
					WaitForCancellation:      true,
					ParentClosePolicy:        enumspb.PARENT_CLOSE_POLICY_REQUEST_CANCEL,
					WorkflowIDReusePolicy:    enumspb.WORKFLOW_ID_REUSE_POLICY_REJECT_DUPLICATE,
				},
			)
 
			futures = append(futures, workflow.ExecuteChildWorkflow(
				child,
				DocumentAnalysisName,
				in,
			))
		} else {
			// Other approved tools run as activities.
			futures = append(futures, workflow.ExecuteActivity(
				workflow.WithActivityOptions(turnCtx, opts),
				WebToolActivity,
				chat.WebToolExecution{Scope: a.Scope},
			))
		}
	}
}

Sources: document_analysis.go · web_approval.go

Parallel Execution

Parallel execution lets a workflow start several independent Activities or Child Workflows before waiting for their results. This can reduce waiting time when one task does not need the output of another. Temporal records their progress, so the workflow can recover after a worker restart without losing track of the scheduled work.

Durable agents can use this pattern to fetch several web pages, analyze different parts of a document, or ask specialist agents to work on separate questions. Remote coding agents can run independent checks, such as linting, type checking, and tests, in parallel. Work that modifies the same files or shared state needs more care and may need to run sequentially.

Practical example: A user uploads a report and asks for a summary. After approval, the demo’s DocumentAnalysisWorkflow prepares the document for analysis. When the document contains multiple chunks, it starts a batch of AnalyzeDocumentChunk Activities without waiting for each one individually.

The batch size is limited by in.Budget.Parallelism. Once all Activities in the batch finish, the workflow moves to the next batch. This keeps the amount of scheduled work bounded instead of submitting every chunk at once. Actual execution speed still depends on worker capacity and the local model server.

Start together, join every resultStatic view
A bounded batch of three document-chunk Activities starts together and finishes at different times. The workflow waits for every result before starting the next batch. Three tasks are illustrative; actual concurrency depends on the budget and worker capacity.
for base := 0; base < result.Chunks; base += in.Budget.Parallelism {
	if ctx.Err() != nil {
		return result, ctx.Err()
	}
 
	// Schedule this batch before waiting for any results.
	var futures []workflow.Future
	for i := base; i < result.Chunks && i < base+in.Budget.Parallelism; i++ {
		futures = append(futures, workflow.ExecuteActivity(
			work,
			"AnalyzeDocumentChunk",
			chat.DocumentChunkInput{
				Analysis: in,
				Index:    i,
			},
		))
	}
 
	// Wait for every activity, requesting cancellation on the first error.
	var first error
	for _, f := range futures {
		if err := f.Get(join, nil); err != nil && first == nil {
			first = err
			cancel()
		}
	}
 
	if first != nil {
		return result, first
	}
}

ExecuteActivity returns a future, so the first inner loop schedules the whole batch without blocking on each Activity. The second loop waits for completion. If it encounters an error, it requests cancellation of the remaining work and still waits for all futures before returning.

Source: document_analysis.go

Approval

The Approval pattern lets a workflow wait for a person’s decision before continuing. Temporal Signals or Updates can deliver that decision without keeping an Activity running or repeatedly polling for an answer. The workflow can recover after a worker restart and continue waiting, subject to its deadline.

Durable agents can use approvals before fetching external content, sending messages, or changing data. Remote coding agents can ask before running a destructive command, pushing changes, or deploying an application. An approval should be tied to the specific action, so an old decision cannot accidentally authorize different work.

Practical example: A user asks to summarize a web page and provides its URL. When the model requests the web_fetch tool, the conversation workflow checks whether permission has already been granted for the conversation’s toolset. If not, it records an approval request and waits for the user’s decision before fetching the page.

Wait for a person, not a timerStatic view
The agent records a permission request for a specific web-fetch action. The workflow waits durably without running the tool. A simulated user approves the action, then the tool proceeds. Real approval must come from the user and remains subject to its deadline.
if s.PendingApproval == nil {
	a.Scope.PolicyRevision = s.Policy.Revision
 
	// Bind the request to this turn, tool call, and policy revision.
	s.PendingApproval = &chat.WebApproval{
		ID: chat.LogicalID(
			s.ActiveTurn,
			"approval/"+a.Scope.ToolCallID+"/"+
				strconv.FormatInt(s.Policy.Revision, 10),
		),
		Scope:    a.Scope,
		Deadline: workflow.Now(ctx).Add(24 * time.Hour),
	}
}
 
p := s.PendingApproval
if p.Scope != a.Scope {
	return temporal.NewNonRetryableApplicationError(
		"unbound pending approval",
		"InvalidRequest",
		nil,
	)
}
 
// Persist the request so the application can show it to the user.
s.Status = "awaiting_approval"
if err := workflow.ExecuteActivity(
	webStorageContext(ctx),
	WebApprovalActivity,
	chat.WebApprovalRecord{Approval: *p},
).Get(ctx, nil); err != nil {
	return err
}
 
// Wait for a decision, a policy change, or a workflow pause.
_, err := workflow.AwaitWithTimeout(
	turnCtx,
	p.Deadline.Sub(workflow.Now(ctx)),
	func() bool {
		return p.Decision != "" ||
			p.Scope.PolicyRevision != s.Policy.Revision ||
			pause()
	},
)

Source: web_approval.go

Long-Running Activity

Some agent operations take longer than a normal web request. A model may spend several minutes generating a response, or a tool may need time to process a large amount of data. Running this work as a Temporal Activity gives it explicit timeouts, retry rules, and a way to report that it is still alive.

Durable agents can use long-running Activities for model generation, document processing, or external jobs. Remote coding agents can use them to run a test suite, build a project, or wait for a sandbox operation to finish. Regular heartbeats help Temporal detect an interrupted worker and deliver cancellation requests.

Practical example: A user asks the demo a question that takes the local Ollama model a while to answer. Model generation runs inside a Temporal Activity, independently of the browser request. While it works, the activity sends heartbeats and the application can stream partial output to the user.

If the user presses Stop, the workflow requests cancellation. If the worker shuts down instead, the implementation treats that as a retryable interruption rather than a user cancellation. Before invoking the model on a retry, it checks whether a complete result has already been saved.

The conversation workflow gives each model attempt a 10-minute execution timeout and requires a heartbeat within 15 seconds. The total activity window, including queueing and retries, is limited to 25 minutes.

Still thinking. Still alive.Static view
A model Activity sends heartbeats even before tokens arrive, streams output, and saves the completed result. Heartbeats report liveness, not a model checkpoint. A retry can reuse a saved complete result; it does not automatically resume generation mid-output.
mctx := workflow.WithActivityOptions(
	turnCtx,
	workflow.ActivityOptions{
		TaskQueue:              WebLLMQueue,
		StartToCloseTimeout:    10 * time.Minute,
		ScheduleToCloseTimeout: 25 * time.Minute,
		HeartbeatTimeout:       15 * time.Second,
		WaitForCancellation:    true,
		RetryPolicy: &temporal.RetryPolicy{
			MaximumAttempts: 2,
			InitialInterval: time.Second,
			NonRetryableErrorTypes: []string{
				"InvalidRequest",
				"ModelNotFound",
			},
		},
	},
)
 
// Identify the model step being executed.
modelRef := ref
modelRef.Step = state.Step
 
var output chat.ModelStepResult
err := workflow.ExecuteActivity(
	mctx,
	WebModelActivity,
	modelRef,
).Get(join, &output)

WaitForCancellation makes the workflow wait for the activity’s cancellation to finish rather than immediately moving on when cancellation is requested. The surrounding implementation also tracks invocation cleanup before allowing replacement work.

Inside model execution, the activity starts a heartbeat helper. This runs independently of token arrival, so the model can remain alive even while it is thinking and has not produced new output.

var tokens atomic.Int64
 
// Report liveness and the current token count while generation runs.
stop := startStreamHeartbeats(ctx, func() {
	if activity.IsActivity(ctx) {
		activity.RecordHeartbeat(ctx, tokens.Load())
	}
})
defer stop()
 
// Reuse a completed result instead of invoking the model again.
if _, err := a.Repository.GetChatModelResult(ctx, ref); err == nil {
	return nil
} else if !errors.Is(err, storage.ErrNotFound) {
	return err
}
 
// ...
 
func startStreamHeartbeats(ctx context.Context, beat func()) func() {
	done := make(chan struct{})
	exited := make(chan struct{})
 
	go func() {
		defer close(exited)
 
		ticker := time.NewTicker(time.Second)
		defer ticker.Stop()
 
		beat()
		for {
			select {
			case <-done:
				return
			case <-ctx.Done():
				return
			case <-ticker.C:
				beat()
			}
		}
	}()
 
	// Stop the helper and wait for its goroutine to exit.
	var once sync.Once
	return func() {
		once.Do(func() { close(done) })
		<-exited
	}
}

Sources: web_session.go · chat_model.go


The following sections are reserved for optimisations with KEDA and Kyverno. These techniques can work alongside the Temporal patterns above, but they are also useful for workloads that do not use Temporal.

Worker autodiscovery with Kyverno

Worker autodiscovery can be a useful addition to the Child Workflows and Parallel Execution patterns. As a system grows, keeping a central list of workers and their dependencies becomes harder. Adding a worker can mean updating both its deployment and the orchestrator’s configuration.

Kyverno is a Kubernetes policy engine that can validate resources and generate other resources from their metadata. This makes it useful for building a lightweight service registry. Services declare their roles through annotations, and policies generate registrations that an application can watch. This pattern works without Temporal too, but the application still needs to understand those registrations.

This setup uses two policies. A ValidatingPolicy checks worker annotations, while a GeneratingPolicy creates a ConfigMap containing the worker’s task queue, workflow type, and dependencies. The orchestrator watches these registrations, validates the dependency graph, and starts Temporal Child Workflows in dependency order.

For an agentic platform, this could register specialist workers for code review, testing, or security checks without maintaining a separate central list. It discovers capabilities declared in annotations, not capabilities inferred from an agent’s code.

The example below shows how a worker Service declares its dependencies:

Declare once, discover automaticallyStatic view
A Kubernetes Service registers declaratively using worker annotations. Kyverno validates the annotations and generates a ConfigMap with the task queue, workflow type and dependencies. The orchestrator watches registrations and validates the dependency graph. Registration does not guarantee worker readiness.
apiVersion: v1
kind: Namespace
metadata:
  name: temporal-demo-worker-d
  labels:
    temporal-demo.io/worker-namespace: "true"
---
apiVersion: v1
kind: Service
metadata:
  name: worker-d
  namespace: temporal-demo-worker-d
  annotations:
    temporal-worker: "true"
    temporal-worker-run-after: "worker-b,worker-c"
spec:
  selector:
    app: worker-d
  ports:
    - name: health
      port: 8081
      targetPort: health

The following shortened policy turns matching Services into registrations:

apiVersion: policies.kyverno.io/v1
kind: GeneratingPolicy
metadata:
  name: temporal-generate-registrations
spec:
  evaluation:
    generateExisting:
      enabled: true
    synchronize:
      enabled: true
    orphanDownstreamOnPolicyDelete:
      enabled: false
 
  matchConstraints:
    namespaceSelector:
      matchLabels:
        temporal-demo.io/worker-namespace: "true"
    resourceRules:
      - apiGroups: [""]
        apiVersions: [v1]
        operations: [CREATE, UPDATE]
        resources: [services]
 
  matchConditions:
    - name: worker-service
      expression: >-
        object.metadata.?annotations[?'temporal-worker']
          .orValue('') == 'true'
 
  variables:
    - name: registration
      expression: >-
        {
          'apiVersion': dyn('v1'),
          'kind': dyn('ConfigMap'),
          'metadata': dyn({
            'name': 'temporal-worker-' + object.metadata.name,
            'namespace': object.metadata.namespace,
            'labels': {'temporal-demo.io/registration': 'true'}
          }),
          'data': dyn({
            'workerName': object.metadata.name,
            'taskQueue': object.metadata.name + '.v1.temporal',
            'workflowType': 'ExampleChildWorkflow',
            'runAfter': object.metadata.?annotations[
              ?'temporal-worker-run-after'
            ].orValue('')
          })
        }
 
  generate:
    - expression: >-
        generator.apply(
          object.metadata.namespace,
          [variables.registration]
        )

Here, CEL expressions replace the earlier {{ request.object... }} templates. Optional lookups with orValue('') handle missing annotations. The dyn(...) casts allow the generated resource to contain values of different types.

generateExisting covers Services that already exist. synchronize keeps registrations aligned with their sources and repairs modified or deleted ConfigMaps. Removing the worker annotation, setting it to "false", or deleting the Service removes its registration. With orphaning disabled, deleting the policy also removes its generated registrations.

This shortened example leaves out the schema-version and source-identity fields required by the demo’s registration contract. Use the full policy when running it.

A separate ValidatingPolicy rejects malformed annotations. Its dependency check uses CEL like this:

# Excerpt: "annotations", "dependencies", and trimmed "tokens"
# are variables defined earlier in the full ValidatingPolicy.
validationActions: [Deny]
validations:
  - expression: >-
      variables.annotations[?'temporal-worker'].orValue('') != 'true' ||
      variables.dependencies.trim() == '' ||
      variables.tokens.all(token,
        token.matches('^[a-z]([-a-z0-9]{0,61}[a-z0-9])?$') &&
        token != object.metadata.name)
    message: Dependencies must contain valid names and no self-reference.

Trade-offs and edge cases:

Kyverno can help with worker and service autodiscovery, but there are limits to what its policies can check.

  • Local validation is not graph validation. The policy checks dependency names, empty entries, and direct self-references, but it cannot validate the full dependency graph. The orchestrator still needs to reject missing dependencies, duplicate worker identities, and indirect cycles such as A → B → C → A. CEL does not turn Kyverno into a recursive dependency-graph engine.
  • Discovery is eventually consistent. Policy reconciliation and Kubernetes watches take time.
  • Registration is not readiness. A registered worker may have no running Pods because KEDA scaled it to zero, as described in the next section. Registration describes how to route work, not whether a process is currently available.

Sources: GeneratingPolicy · ValidatingPolicy · Discovery behavior · Graph validation

Worker scaling with KEDA

Temporal workers are not always busy. Keeping a Pod running for every worker can waste resources, especially when tasks arrive in bursts.

KEDA scales Kubernetes workloads using external signals, such as a queue backlog. Starting with version 2.17, KEDA supports scaling applications based on Temporal task queues.

With Temporal, KEDA watches workflow and activity task queues. When work arrives, it scales up the corresponding worker Deployment. Temporal keeps tasks queued while the Pod starts. Once the queue stays idle and the cooldown expires, KEDA can scale the worker back to zero.

For agentic systems, this is useful for occasional jobs such as repository indexing, code review, or sandbox setup. Workers only need capacity when there is work to process, at the cost of some startup latency.

The ScaledObject below connects worker D’s Deployment to its Temporal task queue:

Queued work brings a worker onlineStatic view
The orchestrator starts a Child Workflow. Temporal queues its first task while no worker is running. KEDA detects the backlog and scales the Deployment from zero to one. Kubernetes creates a Pod. Once ready, the worker polls Temporal and processes the waiting task. KEDA does not consume tasks. Timing is illustrative.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-d
  namespace: temporal-demo-worker-d
spec:
  # Existing Deployment in the same Kubernetes namespace.
  scaleTargetRef:
    name: worker-d
 
  minReplicaCount: 0
  maxReplicaCount: 1 # Demo value, production can use different
  pollingInterval: 5 # Check for work every five seconds.
  cooldownPeriod: 120 # Wait before returning to zero.
 
  triggers:
    - type: temporal
      metadata:
        endpoint: temporal.temporal.svc.cluster.local:7233
        namespace: default # Temporal namespace, not Kubernetes.
        taskQueue: worker-d.v1.temporal
        queueTypes: "workflow,activity"
        activationTargetQueueSize: "0"
        targetQueueSize: "1"
        includeRunningWorkflowCount: "false"
        enableTLS: "false" # Use false for local / demo only

The worker polls worker-d.v1.temporal for both workflow and activity tasks. A backlog above zero activates the Deployment. targetQueueSize sets the backlog target used for scaling.

In the demo’s dependency graph, worker D runs after B and C finish. When the parent starts D’s Child Workflow, Temporal queues its first task. KEDA detects that backlog and starts a worker Pod to process it.

Trade-offs and edge cases:

  • Cold starts add latency. The first task waits for polling, scheduling, image loading, and worker startup. Keep a minimum replica count for latency-sensitive workers, and account for startup delays in task and workflow timeouts.
  • Shutdown and retries still matter. Workers need graceful shutdown, Activity heartbeats, and retry-safe side effects. Temporal preserves workflow history, but an interrupted Activity can run again.
  • Zero Pods does not mean zero infrastructure cost. Temporal, PostgreSQL, controllers, and cluster nodes remain running. Production connections also need appropriate TLS and authentication.

Sources: ScaledObject template · Worker Deployment · Scaling behavior and limitations.