Kubernetes Worker

How it works

When a task is picked up by a Kubernetes worker, it creates a Job with the following pod structure:

1. hessian-init — an init container that downloads your task source code into a shared volume.
2. hessian-deps — an init container that installs Python dependencies (e.g. from requirements.txt).
3. hessian-task — the main container that runs your task handler.

All three containers share an emptyDir volume mounted at /hessian. The init containers use the runner image, while the deps and task containers use the Python image you configure (defaulting to python:3.12-slim).

The worker process itself does not run your task code — it creates the Job, watches the pod lifecycle, streams logs back to the API, and reports the final status.

Agents with provider: claude_code use the same pod layout but with a different deps script and bootstrap. See Claude Code agents for the dedicated pre-baked image and the cold-start trade-off.

Prerequisites

• A running Kubernetes cluster (minikube, EKS, GKE, AKS, etc.)
• kubectl configured with access to your cluster
• A Hessian API key (create one in the dashboard)

Quick start

Start a Kubernetes worker using the CLI:

hessian worker start 
  --type kubernetes 
  --name my-k8s-worker 
  --queue_group <your-queue-group-id>

By default, this will:

• Use your local kubeconfig (~/.kube/config) or in-cluster config if running inside a pod
• Create Jobs in the default namespace
• Use python:3.12-slim as the task image
• Set resource requests to 100m CPU / 128Mi memory
• Set resource limits to 1000m CPU / 512Mi memory

Configuration

All options can be set via CLI flags or environment variables (prefixed with HESSIAN_).

Core settings

Resource limits

Networking and storage

Job lifecycle

RBAC setup

The worker needs permissions to manage Jobs and read pod logs. Create a Role and RoleBinding in your target namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: hessian-worker
  namespace: default
rules:
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "get", "list", "delete", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "get", "watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]

Apply these with kubectl apply -f and then reference the service account:

hessian worker start 
  --type kubernetes 
  --k8s-service-account hessian-worker

Storing your API key as a Kubernetes Secret

Rather than passing your API key as an environment variable, you can store it in a Kubernetes Secret and have the worker inject it into task pods:

kubectl create secret generic hessian-secrets 
  --from-literal=api-key=your-api-key-here

Then reference it when starting the worker:

hessian worker start 
  --type kubernetes 
  --k8s-api-key-secret hessian-secrets:api-key

The API key will be injected into task pods via a secretKeyRef environment variable — it never appears in the Job spec as plaintext.

Running the worker inside the cluster

For production, you’ll typically run the worker itself as a Deployment inside your Kubernetes cluster. This gives you automatic restarts and lets the worker use in-cluster authentication (no kubeconfig needed).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hessian-worker
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hessian-worker
  template:
    metadata:
      labels:
        app: hessian-worker
    spec:
      serviceAccountName: hessian-worker
      containers:
        - name: worker
          image: hessian/worker:latest
          args:
            - worker
            - start
            - --type=kubernetes
            - --name=my-k8s-worker
            - --queue_group=<your-queue-group-id>
            - --k8s-namespace=default
            - --k8s-api-key-secret=hessian-secrets:api-key
          env:
            - name: HESSIAN_API_URL
              value: "https://api.hessian.com"
            - name: HESSIAN_API_KEY
              valueFrom:
                secretKeyRef:
                  name: hessian-secrets
                  key: api-key

Networking considerations

Pod-to-API connectivity

Task pods need to reach the Hessian API to report status and upload artifacts. If the API is not reachable at the same URL the worker uses (e.g., in local development with minikube), set the --k8s-pod-api-url flag:

# For minikube — pods can reach the host via this special DNS name
hessian worker start 
  --type kubernetes 
  --k8s-pod-api-url http://host.minikube.internal:8080

Object storage access

If you’re self-hosting object storage (e.g. MinIO), pods may need a different URL to reach it. Use --k8s-pod-storage-url to rewrite the host in presigned upload URLs:

hessian worker start 
  --type kubernetes 
  --k8s-pod-storage-url http://192.168.49.1:9000

Custom task images

By default, tasks run in the python:3.12-slim image. You can change the default at the worker level:

hessian worker start 
  --type kubernetes 
  --k8s-default-image python:3.11-slim

Individual tasks can also specify their own image in the task definition, which takes precedence over the worker default.

If your images are in a private registry, you’ll need to create image pull secrets and reference them in your worker’s service account, or configure them via the Kubernetes config.

Claude Code agents

Agents created with provider: claude_code run the Claude Agent SDK loop inside a dedicated runner pod, rather than through the in-process LLM loop used by the anthropic/openai providers. The pod layout is the same hessian-init / hessian-deps / hessian-task triple as a regular task pod, but the deps container additionally needs Node 20, the @anthropic-ai/claude-code CLI, and the claude-agent-sdk Python package.

Pre-baked image (recommended)

Installing those dependencies on every cold start adds ~45s of overhead. To skip that, build the pre-baked hessian/claude-agent image and point the worker at it:

# From the root of the hessian repo:
make docker-claude-agent

hessian worker start 
  --type kubernetes 
  --k8s-claude-agent-image hessian/claude-agent:dev

When --k8s-claude-agent-image is set, the hessian-deps container detects the /opt/hessian-prebaked/.prebaked sentinel and copies the baked virtualenv into /hessian instead of running the slow apt/npm/pip path — collapsing the deps phase from ~45s to ~2s.

For production, build the image from your own CI and push it to a registry your cluster can pull from:

docker build -f build/claude-agent/Dockerfile 
  -t ghcr.io/your-org/hessian-claude-agent:v1.0.0 .
docker push ghcr.io/your-org/hessian-claude-agent:v1.0.0

Then start the worker with:

hessian worker start 
  --type kubernetes 
  --k8s-claude-agent-image ghcr.io/your-org/hessian-claude-agent:v1.0.0

Image precedence

Image selection for a claude_code pod’s deps + task containers follows this precedence:

1. A per-agent image set on the agent definition (highest priority — wins over both worker flags).
2. --k8s-claude-agent-image if non-empty.
3. --k8s-default-image (slow path — Node and the CLI install at pod start).
4. Internal fallback: python:3.12-slim (only reached when both flags above are empty).

--k8s-runner-image is unaffected; it only controls the hessian-init container.

Resilience and reconciliation

The Kubernetes worker includes built-in resilience for common failure scenarios:

• Worker restarts — On startup, the worker scans for orphaned Jobs it previously created. It reconciles their status (completed, failed, or still active) and backfills any missed logs.
• Pod failures — OOMKill, eviction, and node failures are detected and reported with specific failure reasons.
• Timeouts — Tasks that exceed their timeout are cancelled, and the Job is cleaned up.
• Log streaming — Logs are streamed in real-time from the pod back to the API, so you can follow task progress in the dashboard.

Full example

hessian worker start 
  --type kubernetes 
  --name production-worker 
  --queue_group qg_abc123 
  --concurrent_task_limit 5 
  --k8s-namespace hessian 
  --k8s-service-account hessian-worker 
  --k8s-runner-image ghcr.io/hessian/runner:v1.0.0 
  --k8s-default-image python:3.12-slim 
  --k8s-cpu-request 200m 
  --k8s-cpu-limit 2000m 
  --k8s-memory-request 256Mi 
  --k8s-memory-limit 1Gi 
  --k8s-api-key-secret hessian-secrets:api-key 
  --k8s-job-ttl 3600