How to Scale Agents with SSH

Distribute agent sessions across a pool of remote build machines so your orchestrator host stops being the bottleneck.

Prerequisites

• A working Sortie setup (the quick start covers this)
• SSH key-based access from the orchestrator host to each build machine (no password prompts)
• The agent binary (e.g., claude, copilot, or codex) installed and on PATH on every remote host
• The standard sleep utility on every remote host, which the remote shell uses to notice that the SSH connection has ended and stop the agent
• The standard dd utility on every remote host, which Sortie uses to deliver environment variables (see give the remote agent its credentials)
• Remote port forwarding allowed by every remote host’s sshd, if agents there should use Sortie’s tools (see let remote agents use Sortie’s tools)
• ~/.ssh/config entries or DNS for your build hosts (recommended but not required)

Verify connectivity before touching any Sortie config:

ssh build01.internal "which claude && command -v dd && command -v sleep && echo ok"

Expected output:

/usr/local/bin/claude
/usr/bin/dd
/usr/bin/sleep
ok

If that fails, fix your SSH setup first. Sortie delegates to the system ssh binary and inherits your full SSH configuration: ProxyJump bastions, FIDO2 keys, agent forwarding all work without Sortie-specific config. The one exception is connection sharing: Sortie turns it off for its own connections, so ControlMaster and ControlPersist in your ssh_config have no effect on them.

Add the worker extension

Open your WORKFLOW.md and add an extensions.worker block to the YAML front matter. List your SSH hosts and set a per-host concurrency cap:

# WORKFLOW.md (front matter excerpt)
extensions:
  worker:
    ssh_hosts:
      - "build01.internal"
      - "build02.internal"
    max_concurrent_agents_per_host: 2

This tells Sortie to run agents on build01 and build02 instead of locally. Each host accepts up to 2 concurrent sessions, giving you 4 total agent slots across the pool. Sortie picks the least-loaded host for each new dispatch.

If you also have agent.max_concurrent_agents set, total concurrency is the lower of the two limits. With max_concurrent_agents: 3 and two hosts at 2 each, you get 3 concurrent agents. The global cap wins.

Give the remote agent its credentials

ssh does not hand your shell’s environment to the remote host, so an agent there starts with whatever that host holds. For the credential its own runtime reads, you need no configuration: every agent kind declares its credential variable names, and Sortie sends those from its own environment on each remote launch.

A claude-code pool picks up the orchestrator’s ANTHROPIC_API_KEY, a copilot-cli pool its GH_TOKEN. An agent-client-protocol pool declares none, so its runtime reads the credential or login the remote host already holds, or a variable you name below. The environment reference has the full per-kind list.

Anything else is opt-in. Name it under ssh_pass_env:

# WORKFLOW.md (front matter excerpt)
extensions:
  worker:
    ssh_hosts:
      - "build01.internal"
      - "build02.internal"
    max_concurrent_agents_per_host: 2
    ssh_pass_env:
      - SENTRY_AUTH_TOKEN
      - NPM_TOKEN

Two things to know before you write that list. It takes variable names, never values. Write - $NPM_TOKEN and the reference resolves while the workflow file is being read, so the entry reaches Sortie holding a token where a name should be; Sortie drops such an entry and warns with its position, never its contents. And a name whose value is unset, empty, or only whitespace is skipped, which Sortie says out loud at startup:

level=WARN msg="ssh_pass_env variable is not set or empty in the orchestrator environment" variable=NPM_TOKEN

Values reach the host on the SSH connection’s standard input, never on a command line, so nothing you carry shows up in ps on the orchestrator or on the build host.

Keep a host’s own login

A carried variable overrides whatever the remote host already holds for that name. That is what you want for a credential you manage centrally, and the opposite of what you want when the hosts sign themselves in.

The case that catches people runs like this. You set tracker.api_key: $GITHUB_TOKEN, which puts GITHUB_TOKEN in Sortie’s environment for the tracker. copilot-cli declares GITHUB_TOKEN as one of its credential names. Now every remote Copilot session authenticates as your tracker token instead of the copilot auth login sitting on the build host. Name it under ssh_disallow_pass_env:

# WORKFLOW.md (front matter excerpt)
extensions:
  worker:
    ssh_hosts:
      - "build01.internal"
      - "build02.internal"
    ssh_disallow_pass_env:
      - GITHUB_TOKEN

Sortie now sends nothing of its own for that name and the host’s login stands. ssh_disallow_pass_env outranks ssh_pass_env, so a name in both lists is not sent either.

This covers what Sortie sends, not what your SSH client sends. A variable listed under SendEnv in your ssh_config that the server accepts under AcceptEnv is forwarded by ssh straight out of Sortie’s environment, and no Sortie setting suppresses it. Drop the SendEnv entry when a variable must not reach the host.

Verify

Restart Sortie and read the startup block. A name you listed that Sortie cannot supply is reported there, before any dispatch. Then dispatch one issue and watch the session reach its first turn: an agent that authenticates is an agent whose credential arrived. A credential that did not arrive shows up as an auth failure from the runtime itself, re-emitted in Sortie’s log under agent stderr.

If the launch fails before the agent runs at all, with this line, the host is missing dd:

level=WARN msg="agent stderr" line="sortie: dd is required on the remote host to receive environment variables"

Sortie needs dd on the host for any launch that carries a variable, which includes every remote opencode launch whatever your configuration says. It ships with coreutils and busybox, so most images have it; distroless and scratch-based images often do not. Install it and the retry succeeds.

Let remote agents use Sortie’s tools

An agent on a worker host calls tracker_api, sortie_status, workspace_history, cost_budget, and notify_operator the same way a local one does. Sortie keeps the tool server on the orchestrator host and gives each remote launch a reverse SSH forward back to it, so the tracker credential never leaves the orchestrator host. This works for every agent kind except mock, and for an agent-client-protocol runtime that accepts HTTP MCP servers. The host only has to allow the forward.

1. Check that the host’s sshd allows remote forwarding for the SSH user. With OpenSSH, AllowTcpForwarding must be yes or remote:

   ssh build01.internal "sudo sshd -T | grep -i allowtcpforwarding"

   If a key in authorized_keys carries the restrict option, add port-forwarding beside it.

2. Request a forward by hand. This uses the same -R form as Sortie, but ExitOnForwardFailure=yes so that a refusal shows up in the exit code; Sortie’s own launch sets no and carries on without the tools:

   ssh -o LogLevel=INFO -o ExitOnForwardFailure=yes -R 127.0.0.1:0:127.0.0.1:9 build01.internal true

   A host that allows the forward prints a line like Allocated port 41873 for remote forward to 127.0.0.1:9 and exits 0. A host that refuses prints Warning: remote port forwarding failed for listen port 0 and exits non-zero. Fix the sshd or key setting from step 1 until the first output appears.

3. Dispatch an issue to a worker host and read the log. A session that reached the tools logs one line:

   level=INFO msg="delivered tool server to session" component=tool-relay

If the host refuses the forward, the session still runs. It has no Sortie tools, Sortie leaves the tool advertisement out of the prompt, and the log says why:

level=WARN msg="declared tool server gap for session" component=tool-relay reason=forward_refused

The session also emits a notification, this session started with a declared capability gap in: tool servers. The other reason values and what each means are in when a session runs without the tools.

Servers from mcp_config

Over SSH, Sortie delivers the HTTP servers from your mcp_config file and leaves out the ones that start a local program (stdio), because the program would have to run on the worker host. Sortie logs each omission:

level=WARN msg="withheld tool servers from remote session" servers=[github]

To keep such a server on remote sessions, run it as an HTTP service the worker host can reach and point mcp_config at its URL.

Update hooks for remote execution

Sortie runs the agent command remotely over SSH, but hooks still execute locally on the orchestrator. When SSH mode is active, Sortie injects SORTIE_SSH_HOST into every hook’s environment with the hostname assigned to that issue.

Your hooks need to use this variable to prepare and clean up remote workspaces. Here is a complete set:

# WORKFLOW.md (front matter excerpt)
hooks:
  after_create: |
    if [ -n "$SORTIE_SSH_HOST" ]; then
      ssh "$SORTIE_SSH_HOST" "mkdir -p \"$SORTIE_WORKSPACE\""
      ssh "$SORTIE_SSH_HOST" "cd \"$SORTIE_WORKSPACE\" && git clone --depth 1 git@github.com:acme/backend.git ."
    else
      git clone --depth 1 git@github.com:acme/backend.git .
    fi
  before_run: |
    if [ -n "$SORTIE_SSH_HOST" ]; then
      ssh "$SORTIE_SSH_HOST" "cd \"$SORTIE_WORKSPACE\" && git fetch origin main && git checkout -B sortie/${SORTIE_ISSUE_IDENTIFIER} origin/main"
    else
      git fetch origin main
      git checkout -B "sortie/${SORTIE_ISSUE_IDENTIFIER}" origin/main
    fi
  after_run: |
    if [ -n "$SORTIE_SSH_HOST" ]; then
      ssh "$SORTIE_SSH_HOST" "cd \"$SORTIE_WORKSPACE\" && git add -A && git diff --cached --quiet || git commit -m 'sortie(${SORTIE_ISSUE_IDENTIFIER}): automated changes'"
    else
      git add -A
      git diff --cached --quiet || git commit -m "sortie(${SORTIE_ISSUE_IDENTIFIER}): automated changes"
    fi
  before_remove: |
    if [ -n "$SORTIE_SSH_HOST" ]; then
      ssh "$SORTIE_SSH_HOST" "rm -rf \"$SORTIE_WORKSPACE\""
    fi
  timeout_ms: 120000

The if [ -n "$SORTIE_SSH_HOST" ] guard keeps your hooks working in both modes. When running locally (no ssh_hosts configured), SORTIE_SSH_HOST is absent and the else branch runs. This means you can test locally and deploy with SSH hosts using the same WORKFLOW.md.

Note the quotes around $SORTIE_WORKSPACE in the remote commands. Workspace paths can contain characters that break unquoted shell expansion.

Start Sortie and verify

Restart Sortie the same way you normally would:

sortie ./WORKFLOW.md

Watch for the SSH mode confirmation in the startup logs:

level=INFO msg="SSH worker mode enabled" host_count=2 max_per_host=2

If you see this instead, something is wrong with your config:

level=WARN msg="max_concurrent_agents_per_host has no effect without worker.ssh_hosts"

That warning means you set max_concurrent_agents_per_host but forgot ssh_hosts, or the YAML nesting is off.

When Sortie dispatches an issue, the logs show which host was selected:

level=INFO msg="workspace prepared" issue_id=42 issue_identifier=PROJ-42 workspace=/tmp/sortie_workspaces/PROJ-42 ssh_host=build01.internal
level=INFO msg="agent session started" issue_id=42 issue_identifier=PROJ-42 session_id=session-abc ssh_host=build01.internal

Monitor host utilization

Sortie exposes per-host usage through two channels.

The state API returns ssh_host on each running session. Hit the endpoint while agents are active:

curl -s localhost:7678/api/v1/state | jq '.running[] | {issue_identifier, ssh_host}'

{"issue_identifier": "PROJ-42", "ssh_host": "build01.internal"}
{"issue_identifier": "PROJ-43", "ssh_host": "build02.internal"}

Prometheus metrics expose a gauge per host:

sortie_ssh_host_usage{host="build01.internal"} 2
sortie_ssh_host_usage{host="build02.internal"} 1

Use this to alert on hosts nearing capacity or to right-size your max_concurrent_agents_per_host setting.

When every host runs as many sessions as the setting allows, eligible issues wait. The dashboard lists them under Not dispatched as “No free SSH host”, and the held list of the state endpoint carries them with the reason ssh_hosts_full:

curl -s localhost:7678/api/v1/state | jq '.held[] | select(.hold.reason == "ssh_hosts_full") | .issue_identifier'

Configure SSH host key checking

Sortie uses StrictHostKeyChecking=accept-new by default: the first connection to a new host accepts its key on trust, and subsequent connections reject key changes. This works for most setups, but your environment may need a different policy.

Add ssh_strict_host_key_checking to the worker block:

extensions:
  worker:
    ssh_hosts:
      - "build01.internal"
      - "build02.internal"
    max_concurrent_agents_per_host: 2
    ssh_strict_host_key_checking: "yes"

If you manage known_hosts externally

Production environments where host keys are baked into VM images or distributed through configuration management (Ansible, Puppet, Chef) should use yes. SSH refuses connections to any host whose key is not already in known_hosts. If someone impersonates a host (MITM), the connection fails.

    ssh_strict_host_key_checking: "yes"

Make sure known_hosts on the orchestrator host contains entries for every host in ssh_hosts before starting Sortie. Missing entries cause immediate connection failures. There is no interactive prompt to accept the key.

If your hosts are stable but you don’t manage keys

Keep the default. Omit the field or set it explicitly:

    ssh_strict_host_key_checking: "accept-new"

The first connection to each host accepts the key automatically. Changed keys are rejected on subsequent connections. This is the current behavior and requires no action.

If your hosts are ephemeral

CI runners, auto-scaled spot instances, and test VMs that get rebuilt frequently reuse IP addresses with new host keys. Use no to prevent known_hosts mismatches from breaking connections:

    ssh_strict_host_key_checking: "no"

For the full list of allowed values, see the worker configuration reference.

Handle SSH failures

SSH connection problems (exit code 255) are transient infrastructure failures. Sortie retries them automatically with exponential backoff. The retry uses host affinity: it prefers dispatching back to the same host, but falls back to the least-loaded alternative if that host is at capacity or unreachable.

A remote “command not found” error (exit code 127) means the agent binary is missing on that host. The run fails with port_exit, its error carrying exit status 127 and the remote shell’s own message, and is retried with exponential backoff, so it keeps failing until the binary is installed. Check that your configured agent.command (e.g., claude, copilot, codex app-server) is installed and on PATH for the SSH user.

A remote agent does not outlive its SSH connection. When Sortie cancels a turn, stops a session, or shuts down, or when the connection drops, the worker host sends the agent SIGTERM and, after agent.stop_grace_ms, SIGKILL. No host cleanup is needed. See remote agent shutdown for the details.

What we configured

You now have a Sortie setup where the orchestrator runs on one machine and agent sessions execute across remote build hosts. The orchestrator handles dispatch, retry, and state tracking. The build machines handle the CPU and I/O of running agents.

The key pieces:

• extensions.worker.ssh_hosts: the pool of remote machines
• extensions.worker.max_concurrent_agents_per_host: per-host concurrency cap
• extensions.worker.ssh_strict_host_key_checking: SSH host key verification policy (accept-new, yes, or no)
• extensions.worker.ssh_pass_env and extensions.worker.ssh_disallow_pass_env: which of Sortie’s own environment variables reach the remote agent, on top of the credential names its agent kind carries by default
• SORTIE_SSH_HOST in hooks: the bridge between local orchestration and remote preparation
• Least-loaded dispatch: Sortie balances work across hosts automatically
• Retry affinity: failed sessions prefer the same host on retry, avoiding redundant workspace setup
• Agent tools: available to every agent kind except mock over a reverse forward, provided each host’s sshd allows it; a session on a host that refuses runs without them
• Token usage: reported remotely on claude-code, codex, and opencode; copilot-cli and agent-client-protocol report none over SSH, because both recover their figures from files the runtime writes on the machine that ran it, which the adapter reads only on a local launch, so token budgets and cost estimates go inert for those sessions

For the full SSH configuration schema, see the WORKFLOW.md reference. For environment variables injected into hooks during SSH dispatch, see the environment variables reference.