Production monitoring#

Inspect the serving release with zyn prod status, or view resource history with zyn prod metrics. Commands use the SSH host in deploy.zon; see Production commands for access setup and all options.

Status#

zyn prod status checks the serving release, HTTP healthcheck, services, host capacity, and TLS expiry. Example output with sample values:

$ zyn prod status
Production  example.com  ● HEALTHY

Serving     blue · release 20260930123456-10 · active 2h 0m
HTTP        2xx /healthz · gateway, worker, DB active
Uptime      host 1d 1h · app 2h 0m · worker 2h 0m

Last hour
  CPU avg   10%   ▁▁▁▂▂▂▁▁▂▁▁▁▁
  Memory    67%   ▆▆▆▆▆▆▆▆▆▆▆▆▆     5.2 / 7.8 GiB
  Disk      43%   ▄▄▄▄▄▄▄▄▄▄▄▄▄     9.3 / 21.6 GiB

TLS         expires in 62d
Deployment  idle

zyn prod status --watch     # Refresh continuously; Ctrl-C stops
zyn prod status --verbose   # Include inactive slots and capacity checks
zyn prod status --json      # Machine-readable snapshot

Healthy status exits 0; degraded or unknown status exits 1. A running service alone does not establish health. Missing metrics history does not make an otherwise healthy application unhealthy.

Metrics#

zyn prod metrics shows the last 24 hours by default. Select a duration from 1m to 1y using m, h, d, w, or y. Example output with sample values:

$ zyn prod metrics 1h
Production metrics: example.com (last hour)

CPU avg    10%   scale 0–40%
Latest:     0.20 / 2.00 vCPU

 40% ┤
     │
     │
     │               ⣀⠤⠊⠑⢄
     │           ⢀⡠⠒⠉     ⠉⠢⡀              ⢀⣀⡀
     │  ⣀⠤⠤⠤⠤⠤⣀⠤⠊⠁          ⠈⠉⠉⠒⠒⠢⠤⠤⣀⣀⡀⢀⡠⠔⠊⠁ ⠈⠉⠒⠢⠤⢄⣀     ⣀⡠⠤⠤⠤⠤⢄
     │⠒⠉                              ⠈⠁            ⠉⠑⠒⠊⠉
  0% ┤
     └────────────────────────────────────────────────────────────
      1h ago        45m            30m            15m          now

Memory     67%   scale 0–90%
Latest:     5.2 GiB / 7.8 GiB

 90% ┤
     │
     │⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉⠉
     │
     │
     │
     │
  0% ┤
     └────────────────────────────────────────────────────────────
      1h ago        45m            30m            15m          now

Disk       43%   scale 0–60%
Latest:     9.3 GiB / 21.6 GiB

 60% ┤
     │
     │⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒
     │
     │
     │
     │
  0% ┤
     └────────────────────────────────────────────────────────────
      1h ago        45m            30m            15m          now

      Blank = no sample (not zero)
      -- = unavailable or stale

zyn prod metrics 7d         # Last week
zyn prod metrics --watch    # Live five-minute view; ←/→ changes the range

Each chart includes used/total capacity for its latest bucket. CPU usage is averaged within the bucket and expressed in vCPUs; memory and disk use the last observation in the bucket, with binary units. These values come from recorded counters, not rounded percentages. A separate DB disk chart appears when recorded disk usage or capacity differs. Older collectors and percentage-only summaries show -- for capacities they did not record; zyn deploy updates the collector.

Blank intervals mean missing samples, not zero usage. -- means the latest value is unavailable or stale. Hosts have no history before the collector is installed; missing readings are not backfilled.

Reference#

Health and status#

Status probes the activated release's healthcheck through its gateway and domain or port, not edited local configuration. Healthy status requires:

open traffic, no pending recovery, known deployment-lock state, and matching slot/release;
running active HTTP, required worker, gateway, and PostgreSQL services;
healthcheck 2xx;
at least 1 GiB free and less than 90% space and inode usage on release and database filesystems;
more than 14 days until certificate expiry when TLS is configured.

An active deployment does not itself make healthy serving services unhealthy; pending activation without a running deployment is degraded. Healthy exits 0; degraded, unknown, SSH, and protocol failures exit nonzero; syntax errors exit 64.

Deployment state is separate: in progress means the host lock is held, pending recovery means activation is pending without an active deployment, and unknown means the lock cannot be inspected. Probes neither wait for nor modify it.

Service uptimes come from systemd; host uptime comes from the kernel. Release activation time is unknown for deployments predating its recording and does not affect health. History covers up to one hour in five-minute intervals; shorter windows show elapsed minutes. Missing/stale history warns without changing health or exit status. Memory usage is total minus available; disk usage is used/total, with the database filesystem separate only when on a different device.

Metrics#

After committing traffic, deployment installs a root-owned one-minute systemd timer and recent sampling service. CPU, memory, and release/database filesystem history lives in /var/lib/zyn/metrics, outside releases and PostgreSQL. Collection survives deployments and rollbacks; setup or sampling failures do not roll back traffic. No public application metrics endpoint is exposed.

History	Retention
About one-second observations	Five minutes
Five-second observations	One hour
Minute samples	90 days
Hourly summaries	One year

Sampling removes expired files. Missing observations are never backfilled; short ranges need the recent collector. Metrics default to 24 hours and accept 1m through 1y using m, h, d, w, or y, including 1h, 24h, 7d, 30d, and 1y. Custom durations use up to 60 buckets; five-second samples approximate unaligned boundaries.

CPU needs consecutive kernel-counter readings: the first after startup/reboot has no percentage. A preceding reading may supply the first in-range baseline only within the same boot and 180 seconds; it is not plotted. Graphs end at command time and show sampled intervals, not live readings. Only adjacent intervals join; missing values are not zero. Identical filesystem usage appears once. Below 36 terminal columns, latest values replace graphs.

Terminal output and watch modes#

One-shot terminal status and metrics show loading views, restore the previous screen on completion, failure, or interruption, and print a completed report once. Redirected/JSON output contains only the final report.

Watch modes use the alternate screen, fit frames without growing scrollback, and print the latest complete report on exit when available. Loading precedes the first report; refreshes retain the previous report. Range changes update headings and axes immediately, never display superseded results, and start collection as soon as the current remote report finishes.

A persistent SSH session collects again one second after each report; slow probes lengthen that interval. Connection/report errors display and retry after one second; degraded health does not stop status watch. Metrics watch works without arrow navigation when input is not a terminal.

Interactive output respects NO_COLOR; logs accept CLICOLOR_FORCE for redirected output. Plain/JSON output has no hyperlink escapes. Headings link configured domains with HTTPS for automatic TLS or HTTP otherwise, without an application port; fallback SSH hostnames are not linked.

Logs reject records over 16 MiB. Repeated, unknown, or missing options exit 64 before SSH; both log commands accept -h and --help. Production commands refuse unmanaged units and never recover deployments, leave maintenance, create configuration, access credentials, build, or refresh .zyn/framework.

Job inspection#

zyn prod jobs status reports pending, running, completed, failed, and discarded counts. --queue filters by strict-snake-case queue; --details adds ready/scheduled counts, expired leases, oldest ready age (milliseconds), and next eligibility (Unix microseconds). Unexpired leases mark nonterminal jobs running; scheduled work and expired leases are pending.

zyn prod jobs failed lists retained failures newest first, breaking ties by UUID. It accepts --queue and --limit (default 20, range 1–1000). show ID reads any retained state; missing IDs fail with JobNotFound.

Inspection exposes identifiers, state, attempt counts, latest error, and UTC times, not arguments, lease tokens, concurrency keys, or credentials. There is no per-attempt history. Errors are limited to 1024 characters with error_truncated indicating truncation; displayed job/queue names are limited to 128/63 characters. Terminal control bytes are escaped.

Job recovery#

zyn prod jobs retry ID       # Failed or discarded jobs
zyn prod jobs discard ID     # Pending or scheduled jobs

Retry resets attempts, terminal timestamps, latest error, and lease state, makes work immediately eligible, and notifies workers. The deployed worker must support the stored name/payload; the command does not compile or inspect handlers. Discard refuses terminal or actively leased jobs and does not cancel running handlers. Conditional updates protect concurrent claims/transitions; missing or ineligible rows fail unchanged with JobNotRetryable or JobNotDiscardable. Only individual UUIDs are supported; there are no bulk or force operations.

Mutations print the operation, UUID, and SSH host to stderr, then require exactly yes from terminal input. Piped input is not consent. --yes skips confirmation; --no-input also requires --yes and still governs SSH trust/authentication. Success prints retried ID or discarded ID to stdout. Connection failure may follow commit: inspect before repeating. Mutations are never automatically retried.

Job commands query the active release's managed PostgreSQL database over SSH; zyn jobs remains local development. Filtered/detailed status, inspection, and mutations require the matching framework schema and restricted application role. Application views, functions, and triggers run with that role, not cluster-admin privileges. Privileged roles, including server-file/program-execution access, are refused. Inspection is read-only; commands leave no persistent helper objects. Mutations take the deployment lock without waiting and refuse active deployment, maintenance, or pending activation. They never recover a deployment or start a worker.

JSON output#

zyn prod jobs status --json emits integer counts:

{"schema_version":1,"command":"production jobs status","counts":{"pending":0,"running":0,"completed":0,"failed":0,"discarded":0}}

zyn prod jobs status --details --json retains counts and adds details: ready, scheduled, expired_leases, nullable oldest_ready_age_ms, and nullable next_eligible_at. Filtering does not change the output schema.

zyn prod jobs failed --json emits schema_version: 1, command: "production jobs failed", and a jobs array. show ID --json uses command: "production jobs show" and one job object. Each object has id, job, queue, state, numeric attempts and max_attempts, nullable last_error, Boolean error_truncated, and UTC available_at, created_at, updated_at, nullable failed_at, discarded_at, completed_at, and lease_expires_at timestamps. Timestamps use YYYY-MM-DDTHH:MM:SS.ffffffZ. An empty failure list succeeds with jobs: []; a missing individual job fails without a JSON document. Mutations do not accept --json.

zyn prod status --json emits schema_version: 2, command: "production status", string host (the configured SSH target), string overall (healthy, degraded, or unknown), string deployment (idle, in_progress, pending_recovery, maintenance, or unknown), and a snapshot object. --verbose may be combined with --json; JSON always includes all snapshot fields. Healthy status exits 0; degraded and unknown status exit 1 with a complete JSON document. SSH failures or invalid remote responses exit 1 with no document and diagnostics on stderr.

All version-2 snapshot values are strings, preserving explicit remote unknown, none, and absent states:

maintenance, pending, release_matches: yes or no.
deploying: yes, no, or unknown, from a nonblocking lock probe.
slot: blue, green, or unknown; release: release identifier or unknown.
gateway, blue, green, blue_jobs, green_jobs, database, edge: systemd active state, absent, or unknown.
host_uptime, gateway_uptime, blue_uptime, green_uptime, blue_jobs_uptime, green_jobs_uptime: decimal seconds, none for an absent or inactive service, or unknown.
activated_at: decimal Unix epoch seconds or unknown.
release_free_kib, database_free_kib: decimal KiB or unknown.
release_space_used, release_inodes_used, database_space_used, database_inodes_used: decimal percentages with a trailing %, or unknown.
health: HTTP status code, 2xx for successful domain probes, 000 for a failed connection, or unknown.
certificate: earliest certificate expiry as decimal Unix epoch seconds, none when automatic TLS is not configured, or unknown.