agent/sandbox Documentation
Docs/Operate
Reference16 min readdocs/TROUBLESHOOTING.md

Troubleshooting

Diagnose common image, mount, terminal, SSH, browser, clipboard, and host-service problems.

View source Markdown

Start here:

bash
ags doctor

Most issues are visible in doctor output.

Large journal, user.log, or syslog containing AGS terminal output#

Older AGS versions inherited Podman's default container logging driver. With journald, interactive terminal output (including repeated TUI redraws) was persisted in the journal under the ags-* container name. Depending on host syslog configuration, it could also be copied into user.log and syslog. This can consume significant disk space and retain sensitive session content.

AGS now launches interactive containers with --log-driver=none. Attached terminal input/output still works, but podman logs no longer retains the session transcript. This does not disable AGS's own stderr diagnostics or Podman's container lifecycle events.

Update AGS, then exit and relaunch existing AGS sessions: the logging driver is chosen when each container is created. An image rebuild is not required. Existing journal/syslog data is not deleted by this change; use your host's log retention and rotation tools to clean it up after preserving needed diagnostics.

error: required mount source missing#

A required mount path in config does not exist.

Fix#

  • Create the path on host, or
  • mark mount as optional = true, or
  • set create = true (for paths safe to auto-create)

Container image missing or outdated#

Symptoms:

  • image-not-found errors
  • old runtime dependencies

Fix#

bash
ags update-image

Then if needed:

bash
ags update-agents

ags update-image failed#

Any failure before publication leaves the working image and its update state unchanged: the error says The existing image and its update state were retained. Read the named component:

  • Rust stable update check failed, rustup …, pnpm …, or RPM update check failed: the upstream metadata or package repositories were unreachable or returned unexpected data. Retry when the network is available. AGS never treats a failed check as "up to date".
  • recorded Fedora base … is unavailable: the recorded base image was removed locally and its digest can no longer be pulled. Run ags update-image --rebase to adopt the current base of the same Fedora release.
  • verified download of <tool> failed: the archive did not match the SHA-256 in the tool lock, or could not be fetched. Re-save the tool selection with ags tools if the lock is outdated.
  • candidate image failed verification: the assembled image did not pass the offline smoke test; the last lines of its output are included. The configured image was not replaced.

If an update was interrupted while publishing, the next ags update-image completes or discards it automatically and reports Recovery: in its summary. If the configured image was changed outside AGS in the meantime, AGS leaves it untouched, clears the interrupted record, and asks you to run the update again.

Update state and the per-image lock live in ~/.cache/ags/image-state/. Deleting this directory is safe: the next update rebuilds its records and reuses every component image that still exists.

Superseded component images become dangling and are not removed automatically. Reclaim the space with:

bash
podman image prune

br / bv / dcg missing inside container#

If catalog-selected sandbox commands are missing or stale.

Fix#

bash
ags tools --packages config/tool-packages.example.json
ags update-image
ags --agent shell -- -lc 'br --version && bv --version && dcg --version'

The example catalog resolves verified archives from upstream releases:

ags tools resolves each selected source into an immutable architecture-specific download lock. Exact-version entries fail if that version is incomplete and never fall forward; latest entries skip immature or incompatible releases and report their rejection reasons.

Startup warning: dcg is missing and Pi/Claude guards will fail closed#

Symptoms:

  • ags doctor reports configured dcg missing inside the sandbox image
  • ags --agent pi or ags --agent claude prints a startup warning about dcg
  • Pi also shows an in-session warning that guarded Bash commands will be blocked

Fix#

bash
ags update-image
ags doctor
ags --agent shell -- -lc 'dcg --version'

If you intentionally want to bypass AGS guard layers for a single Pi/Claude run, use:

bash
ags --agent pi --yolo
ags --agent claude --yolo

psql missing inside container#

If psql is not found in sandbox shell.

Fix#

bash
ags update-image
ags --agent shell -- -lc 'psql --version'

tmux missing when using --tmux#

Symptoms:

  • ags --agent <name> --tmux fails with tmux: not found
  • or AGS prints a message telling you to rebuild the sandbox image

Cause:

  • your existing sandbox image was built before tmux support was added

Fix#

bash
ags update-image
ags --agent shell -- -lc 'tmux -V && test -f ~/.tmux.conf'

Glimpse falls back to Chromium / Chrome detection#

Symptoms:

  • logs mention No Chromium or Chrome installation found
  • sandboxed glimpseui code behaves like ordinary Chromium fallback mode instead of opening a host-owned window

Cause:

  • AGS host UI is disabled or misconfigured
  • your ags binary is stale
  • your sandbox image is stale
  • you are still using an old AGS session

Fix#

Verify your config has a real [host_ui] setup, then rebuild and restart:

bash
cargo build --release -p ags
ags update-image
ags doctor

Start a fresh session, then verify the runtime wiring:

bash
ags --agent shell -- -lc 'echo "$AGS_HOST_UI_SOCK"; echo "$GLIMPSE_BINARY_PATH"; ls -l /opt/ags/glimpse-shim'

Expected:

  • AGS_HOST_UI_SOCK=/run/ags-host-ui/host-ui.sock
  • GLIMPSE_BINARY_PATH=/opt/ags/glimpse-shim
  • /opt/ags/glimpse-shim exists

For a user-facing overview, see docs/GLIMPSE.md.

Mesa warnings overwrite the agent terminal when a popup opens#

Messages such as MESA-INTEL: warning: ... FINISHME come from native graphics libraries used by the host popup renderer, not from Pi's logging. Lowering [host_ui].log_level does not control those messages.

AGS redirects host UI service stderr and inherited renderer stderr to a private host-only log. At startup it prints:

text
[ags] Host UI ready; diagnostics: /run/user/<uid>/ags/host-ui-logs/host-ui-<random>.log

Read that exact path on the host to investigate popup failures. The log remains after the session exits and is not mounted into the sandbox. Memory-corruption messages such as free(): corrupted unsorted chunks still indicate a renderer problem; keeping them out of the terminal is not a fix for the underlying crash.

If output still overwrites the terminal, rebuild the host AGS binary and start a fresh session:

bash
cargo build --release -p ags

This host-side logging change does not require rebuilding the sandbox image. Expected clipboard-client broken-pipe/connection-reset disconnects are handled quietly.

Pi Ctrl-V image paste or /copy clipboard actions fail#

Symptoms:

  • Pi Ctrl-V image paste reports no clipboard data or wl-paste errors
  • /copy reports clipboard failures from inside the sandbox
  • echo "$AGS_CLIPBOARD_SOCK" is empty

Cause:

  • [clipboard].enabled is false or mode = "off"
  • the AGS binary/session predates the clipboard bridge
  • host wl-paste/wl-copy is unavailable or cannot access your host clipboard
  • payload exceeded [clipboard].max_bytes
  • clipboard approval was denied or no dialog renderer was available

Fix#

Verify bridge wiring in a fresh session:

bash
ags --agent shell -- -lc 'echo "$AGS_CLIPBOARD_SOCK"; echo "$AGS_CLIPBOARD_MODE"; echo "$XDG_SESSION_TYPE"; command -v wl-paste; command -v wl-copy'

Expected:

  • AGS_CLIPBOARD_SOCK=/run/ags-clipboard/clipboard.sock
  • AGS_CLIPBOARD_MODE=read or readwrite
  • XDG_SESSION_TYPE=wayland
  • wl-paste/wl-copy resolve to /home/dev/.local/bin/...

If the bridge is intentionally disabled, enable it in config:

toml
[clipboard]
enabled = true
mode = "readwrite"
approval_required = true
approval_seconds = 300

Clipboard reads prompt on the host by default. Enable [host_ui] for the branded dark/light-aware dialog, or install zenity/kdialog for the fallback. If you intentionally want the old session-wide bridge behavior, set approval_required = false.

AGS-launched interactive Pi loads a session-scoped handler that keeps one clipboard paste pending for up to five minutes while you approve, then inserts it exactly once. Additional paste presses during the wait do not duplicate it. Escape/Ctrl-C cancels the pending insertion without clearing the draft; editing/submitting the draft, switching sessions, or opening another Pi prompt also cancels it. Cancelling in Pi does not close a still-open host approval dialog: dismiss it there too. The handler never bypasses approval or automatically sends a message. Its five-minute wait is independent of the configured approval access window.

If the first paste still times out after about three seconds, start a fresh session using a rebuilt AGS binary (cargo build --release -p ags). No image rebuild is needed. The handler applies to AGS's Pi launch, not Pi started manually from a sandbox shell, and requires a CustomEditor-compatible Pi editor. Images remain in private /tmp/ags-pi-clipboard-* directories for submission/draft recovery; temporary files follow the sandbox's normal cleanup lifecycle.

For raw GUI clients, do not rely on clipboard settings; use the explicit --wayland-compositor-passthrough flag only when you intentionally want broad desktop access.

SELinux alerts mentioning pasta and your source tree#

Symptoms:

  • host notifications or logs mention SELinux blocking pasta
  • messages refer to a directory like code or your repo path

Cause:

  • a previously relabeled bind mount left the wrong SELinux context on part of your worktree

Fix#

Restore labels on the affected tree:

bash
restorecon -RFv /home/$USER/code/agent-sandbox

If your repos live elsewhere, run restorecon on the appropriate parent path instead.

missing or unsuitable terminal: xterm-kitty in tmux#

Symptoms:

  • running tmux inside the sandbox fails with missing or unsuitable terminal: xterm-kitty

Cause:

  • your host terminal exports TERM=xterm-kitty
  • the sandbox image needs kitty terminfo entries available for tmux to start cleanly

Fix#

Rebuild the image so the sandbox includes kitty-terminfo:

bash
ags update-image
ags --agent shell -- -lc 'echo "$TERM" && tmux -V'

Temporary workaround if you need it before rebuilding:

bash
TERM=xterm-256color ags --agent shell

Cannot reach host service from agent (localhost confusion)#

Symptoms:

  • Service works on host but fails from inside agent/shell
  • curl http://localhost:<port> fails in sandbox

Cause:

  • Agent runs inside container, so localhost points to container itself.

Fix#

Use host gateway name instead:

bash
curl http://host.containers.internal:<port>

Example:

bash
ags --agent shell -- -lc 'curl http://host.containers.internal:3000/health'

You can verify runtime hint env vars too:

bash
ags --agent shell -- -lc 'echo "$AGS_HOST_SERVICES_HOST" && echo "$AGS_HOST_SERVICES_HINT"'

Agent CLI missing inside container#

If command like pi, codex, gemini, opencode, or claude is missing/old.

Fix#

bash
ags tools --packages config/tool-packages.example.json
ags update-agents

Ensure the agent is selected in the Agent CLIs panel. A disabled agent is rejected before container startup even if an old launcher still exists in a cache volume.

Agent sessions and updates#

ags update-agents installs and verifies an isolated candidate. If its runtime manifest matches the current generation, it reports Already up to date, discards the candidate, and keeps current/previous unchanged. Otherwise it shares identical immutable files and selects the new generation for new sandboxes. Installer work still occurs on no-op checks; this is not a registry-only preflight. Running and stopped containers keep their concrete runtime mounts; the command reports their generation references using Podman inspection. Cleanup retains the latest, previous, and all container-referenced generations, plus any generation leased by a pending launch. Other completed generations are automatically removed after a successful update. Legacy runtime directories are also removed once no running/stopped container or updated-AGS launch lease references them. Ordinary update failures immediately discard their own unselected candidate and sweep older stale candidates. Candidates left by crashes are removed after a 10-minute grace period only when no container references them and no installer/verification lease is held. General caches remain untouched. Upgrade the host AGS command before updating; old launchers cannot coordinate with cleanup or use deleted legacy installs.

If an older in-place update already broke a session, generation-based updates cannot repair that process: restart its sandbox after updating. New agent processes inside an existing sandbox still use that sandbox's pinned generation. Use a new sandbox for the new version.

If the updater cannot inspect containers or acquire the update lock, resolve the Podman error or wait for the other update; do not delete lock files. If inspection fails during post-update cleanup, the update remains published but cleanup is skipped with a warning. A failed install or version check leaves the previous selection intact. See runtime generations for retention and disk usage details.

Workspace cache pruning reports orphans but removes nothing#

text
26 orphaned; 0 removed; 0 pending; 0 filesystem deletions

This is normal on a first run of ags prune-workspace-caches: it starts the observed-orphan grace period, measured from first detection, not cache creation. The systemd service uses one day (--grace-days 1); the CLI default remains seven days. Dry runs do not start that clock. pending means a tree has already been quarantined and only partially deleted; it does not count caches waiting for grace. After grace expires, running or stopped containers and pending launch leases can still retain an orphan.

To clean sooner, explicitly review candidates and bypass only grace:

bash
ags prune-workspace-caches --dry-run --grace-days 0
ags prune-workspace-caches --grace-days 0

Protection checks remain enabled. Each run processes at most two cache trees and 1000 filesystem deletions by default, so large caches may take many daily runs. Use --max-caches/--max-deletions only if you accept more work per invocation. Never manually remove mounted caches or their lease/lock files. Valid checkouts, inaccessible/ambiguous identities, and unrelated cache domains are retained; this command is not a general disk purge.

Daily workspace-cache timer is not running or cleanup failed#

The optional scheduler is a systemd user timer, not a root/system service. Inspect it in the same user's session:

bash
systemctl --user is-enabled ags-prune-workspace-caches.timer
systemctl --user list-timers ags-prune-workspace-caches.timer
systemctl --user status ags-prune-workspace-caches.service
journalctl --user -u ags-prune-workspace-caches.service -n 30
systemctl --user cat ags-prune-workspace-caches.service ags-prune-workspace-caches.timer
  • Unit not found: ags install does not install a scheduler. Follow systemd installation.
  • Executable/config missing: check command -v ags and the service's ExecStart. The template assumes ~/.local/bin/ags; source installations commonly use ~/.cargo/bin/ags. Maintenance intentionally does not create missing configs or read repository overlays.
  • No run while logged out: check loginctl show-user "$USER" -p Linger. A persistent timer catches up missed runs when the user manager starts, but running while logged out requires lingering/user-manager availability.
  • Next run is later than 03:17: a random delay of up to 15 minutes is intentional. The schedule uses local time.
  • Podman inspection failure or five-second deadline: verify Podman works in the user's service environment and uses the same connection/storage context as normal launches. Inspection fails closed; no eligible cache is deleted from that attempt. Do not switch to root or another Podman context to bypass it.
  • Service timeout or pending trees: the 60-second service limit and deletion budgets keep impact low. Quarantined trees are resumable on subsequent runs; interruption does not require manual deletion.
  • Priority-policy error: nice/idle-I/O setup must succeed before maintenance mutates caches. Check kernel or sandbox restrictions rather than bypassing the safety/impact policy.

Only enable one scheduler: the daily systemd timer or the daily cron alternative, not both. To stop scheduled runs, use systemctl --user disable --now ags-prune-workspace-caches.timer.

A pinned pnpm version is unavailable or unsuitable#

[sandbox].pnpm_version accepts "latest" (the default) or an exact stable version such as "12.9.1". Ranges and prereleases fail config validation. If a pin is unavailable or its integrity cannot be verified, the image update fails and retains the existing image; it never substitutes latest.

Choose another exact stable version, or remove the setting/set it to "latest", then run ags update-image. A pin controls the shared image, so use distinct image tags for configurations needing different versions. Pinning an older release can reintroduce upstream bugs, including read-only global-directory permission checks during agent verification.

Project pnpm reports ERR_PNPM_UNEXPECTED_STORE after upgrading#

Normal sandboxes now use a writable per-worktree store at /var/cache/ags/pnpm/store; /usr/local/pnpm is reserved for the immutable managed-agent runtime. Existing node_modules may record the former /usr/local/pnpm/.store location.

Stop processes using that worktree, then reinstall its project dependencies using the project's normal pnpm command. AGS intentionally does not rewrite pnpm bookkeeping, delete node_modules, or alter lockfile integrity during launch. Other worktrees have separate caches and need migration only when used. Run ags update-image to install the current stable pnpm release and image defaults; launch-time configuration supplies the storage split for new AGS sessions.

pnpm reports ERR_PNPM_UNEXPECTED_STORE, MODULE_NOT_FOUND under /usr/local/pnpm, /usr/local/dist/pnpm.mjs is missing, or Pi loads from .npm-global#

Cause:

  • a pnpm self-update or install script wrote pnpm's own shims into the persistent agent runtime volume
  • an older sandbox image copied the pnpm launcher instead of preserving its package-relative symlink
  • old npm-global agent installs can shadow the pnpm-managed AGS agent shims
  • those shims can disagree with the store used by the globally installed agent CLIs

Fix#

bash
ags update-image
ags update-agents

With pnpm 11, Pi and Gemini launchers are in /usr/local/pnpm/bin. OpenCode is installed outside pnpm at /opt/opencode-home/.opencode/bin/opencode in its dedicated persistent opencode-install volume. Codex uses its official standalone installer and remains at /usr/local/pnpm/codex. Runtime pnpm comes from the sandbox image at /usr/local/bin/pnpm, so stale persistent shims cannot shadow it. update-agents builds a fresh runtime generation rather than modifying existing installs. Start a new sandbox to use it; running sandboxes retain their original runtime. Legacy shared npm-global files are not removed automatically.

SSH problems (git auth/signing)#

Symptoms:

  • cannot push
  • signing fails
  • keys not loaded

Fix#

  1. Re-run setup:
    bash
    ags setup
  2. Confirm public keys are added in GitHub:
    • auth key as SSH key
    • signing key as SSH signing key
  3. Re-run:
    bash
    ags doctor

If keys are passphrase protected, ssh-add may prompt interactively.

Secret not available inside container#

Symptoms:

  • tool auth failures
  • missing token env vars

Check#

  • Is [[secret]] / [[tool.secret]] configured correctly?
  • Is source env var actually set and non-empty?
  • If using secret_store, does secret-tool lookup ... return a value?
  • If using command, is the executable present and executable on the host?
  • Does the helper finish within five seconds, exit 0, and print exactly one non-empty UTF-8 value (optionally followed by one \n or \r\n)?
  • Is the command declared in the user/global config? Repo-local .ags/config.toml command sources are rejected.
  • Run ags doctor; it checks command availability and lookup success without printing the resolved value or helper stderr/stdout.

Fix#

  • Re-run ags setup to re-enter secrets (if using interactive keyring flow)
  • Export env vars before launching ags
  • Run the configured helper directly on the host to diagnose native credential-store access. Avoid sharing its output in logs or bug reports.

Command helpers receive only PATH, HOME, USER, LOGNAME, DBUS_SESSION_BUS_ADDRESS, and XDG_RUNTIME_DIR when those variables exist. If a helper depends on other ambient host variables, wrap it in a narrowly scoped adapter that establishes only the required context. Resolved values are delivered as ordinary container environment variables and can be inspected by sandboxed processes.

Browser mode fails#

Symptoms:

  • --browser exits early
  • debug endpoint not reachable

Check#

  • [browser].enabled = true
  • [browser].command exists and executable
  • [browser].profile_dir is valid
  • [browser].profile_dir is writable; AGS creates an isolated session profile and discovers the OS-assigned debug port from Chromium's DevToolsActivePort file
  • [browser].startup_timeout_ms is high enough for the host; try 30000 on slow or heavily loaded systems

Fix#

bash
ags doctor
ags --agent pi --browser

If needed, try a different debug port.

Auth proxy: URL opens are auto-denied#

Symptoms:

  • Agent tries to open a URL (e.g. OAuth login) but nothing happens
  • Shim logs show prompt denied

Cause:

  • No AGS dialog renderer is available, so the proxy cannot show the allow/deny dialog and defaults to deny.

Fix#

Enable the branded Glimpse-backed host UI dialog via [host_ui], or install a fallback dialog tool:

bash
# Debian/Ubuntu (GNOME)
sudo apt install zenity

# Fedora (GNOME)
sudo dnf install zenity

# KDE
sudo apt install kdialog   # or: sudo dnf install kdialog

Auth proxy: socket connection failures#

Symptoms:

  • Shim prints failed to connect to auth proxy socket or times out
  • OAuth flows hang

Cause:

  • The per-run runtime directory or socket was not created (e.g. $XDG_RUNTIME_DIR is missing or not writable).

Fix#

Verify your runtime dir:

bash
echo "$XDG_RUNTIME_DIR"
ls -la "$XDG_RUNTIME_DIR"

If unset/missing, set it before running ags:

bash
export XDG_RUNTIME_DIR="/run/user/$(id -u)"

MCP server re-authenticates every session#

Symptoms:

  • MCP servers (e.g. Linear) ask to re-authenticate on every ags run
  • OAuth tokens are not persisted

Cause:

  • MCP auth tokens are stored in ~/.mcp-auth inside the container, which is ephemeral by default.

Fix#

Mount the token directory from your host:

toml
[[mount]]
host = "$HOME/.mcp-auth"
container = "/home/dev/.mcp-auth"
mode = "rw"
kind = "dir"
create = true

To also skip the allow/deny dialog for known providers:

toml
[auth_proxy]
auto_allow_domains = ["mcp.linear.app"]

Auth proxy: OAuth callback not received#

Symptoms:

  • Browser opens the OAuth provider page and redirects to localhost:<port>/callback
  • But the agent inside the container never receives the auth code

Cause:

  • The callback port was already in use on the host when the proxy tried to bind it.
  • Or the OAuth provider uses a non-standard callback parameter name that the shim does not detect.

Fix#

  • Retry the flow — the proxy binds the callback port dynamically per session.
  • If persistent, check for port conflicts: ss -tlnp | grep <port>.
  • The shim detects redirect_uri, callback_url, return_url, and redirect_url query parameters. Other parameter names are not recognized and will fall back to a simple (non-callback) URL open.

Config parse/validation errors#

Symptoms:

  • startup fails with validation message

Fix#

  • Compare your file with config/config.example.toml
  • Verify enums are valid:
    • mount mode: ro|rw
    • mount kind: dir|file
    • mount when: always|browser
  • Verify required strings are non-empty

ags command not found#

If installed from source but not in PATH.

Fix options#

Run via cargo:

bash
cargo run -p ags -- --agent pi

Or self-link:

bash
cargo run -p ags -- install --link-self

Ensure ~/.local/bin is in PATH.

Alias/wrapper commands not found#

If you ran ags create-aliases but short names are missing.

Check#

  • Wrappers mode writes to ~/.local/bin
  • Aliases mode updates shell rc and requires shell reload

Fix#

bash
ags create-aliases --mode both
exec $SHELL

Podman runtime issues#

Symptoms:

  • podman command failures
  • permission/network oddities

Check#

  • Podman installed and working rootless
  • user session has required Podman setup

Run a quick check:

bash
podman info

Then rerun:

bash
ags doctor

Prepare hook refusal or failure#

  • New/changed hooks require an interactive AGS-owned approval of host-user execution. Rerun with the same config, or use ags hooks test NAME --config PATH to review one hook. Test executes host code; it is not an inert dry run. Repository overlay approval must be granted separately before project hook hash approval.
  • Use ags hooks validate response.json to check a response without execution. Stdout must be one JSON response, not diagnostics; send diagnostics to stderr.
  • Timeout/output limits, nonzero exit, reserved env keys or protected/overlapping paths fail closed and are not retried. Pick independent data destinations, not nested overlays into existing workspace or AGS-managed mounts.
  • Native $ORIGIN libraries and script $0 resource discovery may need explicit absolute paths: execution uses a private approved-byte copy. AGS never falls back to the mutable original entrypoint.
  • Install/update an op supporting inject for surviving hook references. Host op authentication is required only at the final launch handoff, never for schema, validation or the redacted hook test. In lockdown, remove/override surviving refs and host mounts rather than expecting them to be silently ignored.

See Startup hooks for trust, process/resource ownership and temporary secret-transport limitations.