uncloud

This release brings easier debugging of failed deploys, new troubleshooting commands (uc machine logs and uc machine rtt), and a nightly release channel.

Read on for the full list of changes and upgrade instructions.

Show logs from a failed deploy

Changes: 8afe523, 303c8e4, 56a54ab

When uc deploy fails because a pre-deploy hook failed or a new container couldn't become healthy, it now prints the last 10 (configurable) log lines from the failed container.

This is a quality-of-life improvement that saves you from running a follow-up uc logs to figure out what went wrong.

See Failed container logs for more details.

Filter service logs by container

Change: 2045819

uc logs now accepts a SERVICE/CONTAINER form where CONTAINER is a container name, full ID, or unique ID prefix. This is handy when one replica of a service is misbehaving and you want to look at it without the noise from the others:

uc logs web/2f60
# Mix and match
uc logs api/61d57fd3428f web/2f60 db

Stream machine logs

⚠️ This requires both uc and the daemon to be upgraded to v0.19.0

PRs: #282, #283. Thanks to @miekg for the contribution ❤️

You can now stream logs from the systemd services that run Uncloud itself on remote machines using the new uc machine logs command. It covers three main services:

• uncloud - the Uncloud daemon
• uncloud-corrosion - the Corrosion service providing the distributed cluster store
• docker - the Docker daemon

For example, stream the Uncloud daemon logs from all machines with:

uc machine logs -f uncloud

This is useful for troubleshooting Uncloud operations without having to SSH into every machine and run journalctl yourself.

See uc machine logs for more details and examples.

Round-trip time between machines

⚠️ This requires both uc and the daemon to be upgraded to v0.19.0

PR: #226. Thanks to @jabr for the contribution ❤️

The new uc machine rtt command shows the round-trip time between every pair of machines in the cluster. This gives you a real-time view of how the mesh network is performing without needing to run manual ping tests.

The data is collected from Corrosion's gossip protocol, which samples latency between peers as part of its normal operation.

$ uc machine rtt
MACHINE     PEER        MEDIAN   STDDEV
machine-1   machine-2   140ms    ±19.4ms
machine-1   machine-3   39ms     ±1.1ms
machine-2   machine-1   168ms    ±18.5ms
machine-2   machine-3   203ms    ±42.3ms
machine-3   machine-1   40ms     ±2.0ms
machine-3   machine-2   158ms    ±15.2ms

A new RTT column has also been added to uc wg show to show the median round-trip time to each WireGuard peer.

Local machine upstreams first in Caddyfile

Change: 45cf87a

The generated Caddyfile now lists upstreams from the local machine first for each service.

On its own this doesn't change routing behaviour (Caddy's default random load balancing policy ignores order). But if you pair it with the first policy in a custom Caddy config, you can always prefer the same-host replica and only fall back to remote machines when the local one is unhealthy.

services:
  app:
    ...
    x-caddy: |
      example.com {
          reverse_proxy {{upstreams 8000}} {
              import common_proxy
              lb_policy first
          }
          log
      }

This saves a cross-machine WireGuard hop for every request that hits Caddy on the machine which already has a replica of the target service. This is especially useful if you have a multi-region setup.

Nightly builds

PR: #308. Thanks to @tonyo for the contribution ❤️

Every push to main now produces a set of nightly binaries tagged as the nightly release on GitHub.

They're great for testing upcoming unreleased changes and reporting regressions early. They might be unstable, so please don't run them in production.

Install the nightly uc CLI locally:

curl -fsS https://get.uncloud.run/install.sh | VERSION=nightly sh

Initialise a cluster or add a machine with a nightly daemon:

uc machine init --version nightly user@host
uc machine add --version nightly user@host

Improvements

• uc machine init/add now embed the install script into the uc binary and send it to the remote machine over the existing SSH connection instead of curl | bash (fe829ef).
• New uc ctx show command prints the name of the currently active cluster context, great for shell prompts and scripts (#317).
• SSH connections now use -o StrictHostKeyChecking=accept-new, which silently accepts host keys on first connection but still protects against key changes later (#303).
• The STORE column in uc images is hidden when all machines use the containerd image store, which is the default for new clusters (c38d916)
• Image push errors now include the underlying error from the unregistry proxy, which makes it much easier to tell apart a broken image push from a broken SSH tunnel (d340659)
• ucind now accepts a UNCLOUD_CONFIG environment variable to override the config file path for development clusters (#315).
• New ucind cluster ls command lists the local development clusters managed by ucind with their machines (#316).

Bug fixes

• Fixed containerd socket auto-detection for unregistry after a machine reboot (47199c3).
• Fixed Caddy config being regenerated on every service change because container records were serialised non-deterministically (#111).
• Fixed SSH control socket path on WSL2 when the runtime directory doesn't exist (#319).
• Fixed a stale SSH ControlMaster connection sometimes causing uc machine init/add to hang (fdbffbf).

Upgrade to 0.19.0

Uncloud CLI locally

To upgrade the Uncloud CLI (uc) locally:

# Homebrew (macOS, Linux)
brew upgrade uncloud

# Install script (macOS, Linux)
curl -fsS https://get.uncloud.run/install.sh | sh

Machine daemon

To upgrade the Uncloud daemon on your machines, run the following commands on each machine:

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -fsSL -o uncloudd.tar.gz https://github.com/psviderski/uncloud/releases/download/v0.19.0/uncloudd_linux_${ARCH}.tar.gz
tar -xf uncloudd.tar.gz
sudo install uncloudd /usr/local/bin/uncloudd
rm uncloudd uncloudd.tar.gz
sudo systemctl restart uncloud

Changelog

• 4d5ca42a9d50c2ff653c8bd9e9bb5afc71663441 chore: add more info about cluster connections in error for 'machine rm'
• f60a9ff293153be3e5d43d3286b959aca5b8ce6a chore: check SSH TCP forwarding for dial operations and return a friendlier error
• 3310b478a1840741c0a19fbf00c864a5ca410ac6 chore: do not log too noisy 'Sent log stream heartbeat.' log line for machine logs
• d340659370ae12641939617c88133f6727879384 chore: enrich image push errors with errors from proxy to unregistry
• c38d916dbbbc411d435c11fa11fc462c27361d9a chore: hide STORE column in 'uc images' output if all machines use containerd image store
• f41590157fd33229e43f3bdedd34967117d939f4 chore: make experiments a separate Go module, remove unnecessary dependencies from root module
• 39181708dda3f9e85c1eee106730b3f225d0a718 chore: set x-context for website deploy
• 7b554cd703d3e2eee96a62709548cd551836c696 chore: style the machine reset prompt in red and fix the padding for [y/N]
• 208a561003507b3faea199364853ee1500c6bfac chore: trigger container sync on ActionHealthStatusRunning Docker event as well
• dd3d809c6effab3aa391f810e9baddf96887aad5 chore: use a non-registry image format for website
• 60ff0889719958e214d7b7365a64027859c59d37 ci: Add go build/module caching
• 02a318cd2ed8fb54f9c09e6016abcb2365f809ca ci: Build and publish latest (nightly) binaries (#308)
• b1be80adf23ade8fa72719ac5c03b20fc901cbda ci: Run nightly builds on macos runners
• 9cfcf49d2855a1f63d081253d2a2a3afff3754ff ci: Update cache key for nightly builds
• 45cf87a0cb6e11e5a5f9cb4ee4e840811d6e1a1e feat(caddy): order local machine upstreams first in generated Caddyfile
• 8afe52367fa40f78b67c77c606d15eda204bb06f feat(deploy): print last logs from failed pre-deploy hook
• 303c8e4506d5f57dcb75126e58c52f69d986f785 feat(deploy): print last logs from new container when fails to become healthy
• f44ada06b91f0fbe897b1234f9c91466dc4652d9 feat(logs): 'uc machine logs' to view logs from systemd services on machines (#283)
• 56a54ab3fd56e83e6265c8dcbb57faecd86f2d93 feat(logs): include hook information in log entry formatting
• 204581973565e8249f489b3d4d662f2edeb638e5 feat(logs): update 'uc logs' command to support filtering by service/container
• 8d023f5c53abc37235ed47f00297cf98d203179b feat(rtt): 'machine rtt' command to show round-trip time between macines usign using gossip data (#226)
• 6409a47c718b002363f7290e54f871c9e98e055e feat: add uc ctx show command to print the current cluster context (#317)
• e8111a419f061db114f78a3fb0f05273b6591e5e feat: auto-accept only new SSH host keys using "-o StrictHostKeyChecking: accept-new" (#303)
• dd989a281e6ddb4e8e7469643de299a0a0599e0e feat: check automatically if uc can connect via Unix socket when running on cluster machine (#296)
• 1c0d48cb46fc75ecdf169da4b254d93cf6e17873 feat: ucind: allow overriding the config.yaml via env var (#315)
• 7e1b91c3725419cdd67aa66586a566af48259066 feat: ucind: implement cluster list command (#316)
• 9ce624facd2a5ba16f51517d64be87017c1bd01c fix(logs): print logs with zero timestamps immediately to prevent indefinite stalling
• b1897ce7199373c3fdddfb55c4bcded6655110b1 fix(nightly): inject correct semver 0..0-nightly-abc1234 for nightly builds
• 6e9acefbe6e7920df8c0c7e2a5af6614020e18fd fix: Caddy config regeneration due to non-deterministic container serialisation (fixes #111)
• 3a79aeffccb9151af339e5e85bfe6f6a23967aab fix: SSH control socket path in WSL2 when runtime dir doesn't exist (fixes #319)
• f8234916cb3910b375b5de6d88120e0bce34f8af fix: another attempt to fix flaky e2e tests: bump dind and get rid of incomplete cgroups fix, pre-allocate ucind machine port, allow ucind container retarts
• fdbffbf020a584a970654432b0137db4d46c2800 fix: close stale ControlMaster ssh connection for machine init/add
• 47199c36ceaa760868e46a51c5caca9afdc9086b fix: containerd socket detection and unregistry startup on machine reboot
• 6fb68c22c5be98fa93fe4a48886a93086555567c fix: correctly wait for journalctl processes to not leave zombies when streaming machine logs (#325)
• 50057c42e06b758ced413d665842e8dd9c3f3483 fix: machine logs timestamp parsing and streaming for systemd <v255
• c888e0a76f7c65af76c6ecbde2ce82a06b8e9846 fix: probe for checking TCP forwarding over SSH (fixes #321)
• 102421df19dcc6f3b28ff141d8deeb97b13fe8d4 fix: tests after DinD update
• 8384d4162cf6fa9d04d3d5b5fd83ff7aa4ea1fa0 lint
• 5b5cd44d680ff040c0ea88785114ac8e6fba77a3 refactor(logs): encapsulate printing errors in PrintEntry
• b852d34068bbf55dcb17ee625191acde0b46dbd9 refactor(logs): update log formatting for systemd services and handle merging in the client
• 84990ad692d1839243ab40d792afdf5dc6a7d945 refactor(rtt): use proto Duration, add tests
• fe829efbf58571aa86ae487392c8656a181dfdbf refactor: embed install.sh script in uc CLI to not curl | sh and version together with CLI
• 19c13f6cd2b1f783f14ff8762cd133c2fef4aec9 refactor: move internal logs pkg from cmd to internal/cli
• 0f5f8727a3352c53167705cabda951f5a5f15af0 test(logs): add unit tests for ParseServiceArgs function